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Higher  education's  increased  use  of  performance-based  funding  (PBF)  and 
performance-based  budgeting  (PBB) — linking  funds  to  institutional  performance  on 
quantitative  or  qualitative  criteria — created  debate  in  two  areas:  the  selection  of  valid 
performance  indicators  (Pis)  and  the  use  of  those  indicators  for  institutional  comparisons 
and  fianding.  Institutional  performance,  as  most  often  defined  in  the  performance- 
funding  arena,  relied  on  easily  measured  institutional  graduation  rates  and  transfer  rates. 
The  problem  with  most  systems  of  performance-based  fiinding,  which  relied  on  these 
absolute  performance  indicators,  was  that  they  failed  to  capture  the  large  diversity  of 
community  college  student  intents,  abilifies,  goals,  and  outside  responsibilities. 

This  dissertation  worked  toward  a  value-added  model  which  could  provide  a  cost- 
effective  way  to  accommodate  and  reward  a  substantially  expanded  range  of  learners  and 
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student  goals  while  still  making  funding  dependent  on  demonstrated  outcomes.  Instead 
of  reliance  on  absolute  indicators  of  institutional  performance,  such  as  graduation  or 
transfer  rates,  the  value-added  approach  established  an  appropriate  reference  norm  by 
accounting  for  the  diverse  backgrounds  and  intents  of  first-time  college  freshmen  at 
community  colleges.  '  . 

The  logistic  regression  analysis  utilized  in  this  study  resulted  in  a  model  that 
included  two  independent  predictor  variables  for  the  outcome  variable:  graduation  or 
transfer.  The  model  predicted  the  graduation  or  transfer  rates  of  the  group  sampled  with 
approximately  62  percent  accuracy.  A  more  precise  model  should  be  developed  before  it 
is  used  for  performance  based  funding  or  performance  based  budgeting  purposes. 


CHAPTER  1 
INTRODUCTION 


In  response  to  state  policymakers'  interest  in  the  accountability  and  performance 
of  higher  education  institutions,  many  community  colleges  found  that  performance 
assessment  policies  of  the  mid-1990s  inadequately  rewarded  this  segment's  contribution 
to  postsecondary  education.  Performance-based  funding  (PBF)  and  performance-based 
budgeting  (PBB) — linking  fimds  to  institutional  performance  on  quantitative  or 
qualitative  criteria — created  debate  in  two  areas:  the  selection  of  valid  and  reliable 
performance  indicators  (Pis),  and  the  interpretation  and  comparison  of  such  measures  of 
success  across  institutions  and  systems. 

Using  absolute  performance  indicators  for  the  assessment  of  postsecondary 
institutions  presented  difficulty  because  Pis  could  not  be  accurately  interpreted  without 
knowledge  about  the  type  of  students  admitted  by  individual  institutions.  As  Astin  wrote, 
"Such  quality  judgments  [were]  meaningless  without  'input'  data  on  the  students  when 
they  first  enrolled]."'  He  argued  that  "'outcome'  scores  [such  as  graduation  and  transfer 
rates]  [were]  probably  telling  us  much  more  about  the  population  recruited  by  the  school 
than  they  [were]  about  the  effectiveness  of  the  school's  academic  program."'^ 

Purpose 

The  purpose  of  this  study  was  to  examine  the  use  of  performance  assessment  and 
performance-based  fiinding  and  budgeting  (PBF&B)  in  United  States  community 
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colleges.  This  study  was  designed  to  (1 )  determine  whether  current  models  and  Pis  used 
for  PBF&B  were  appropriate  for  use  in  two-year  postsecondary  institutions  and,  if  not,  to 
(2)  develop  a  predictive  model  of  community  college  student  performance  which  could 
be  used  as  Pis  for  the  assessment  of  community  colleges.  This  analysis  used  a  database 
containing  longitudinal  data  collected  from  a  nationally  representative  sample  of 
community  college  students.  The  method  used  to  address  this  question  was  first,  to 
determine  the  Pis  most  commonly  used  for  the  assessment  of  four-year  postsecondary 
institutions,  and  then,  to  develop  a  model  for  adapting  those  indicators  to  the  two-year 
postsecondary  sector.  Did  the  most  commonly  utilized  Pis  accurately  assess  the 
performance  of  two-year  postsecondary  institutions? 

Research  Question  One 

Performance  Assessment  and  Performance  Measures 
for  the  Allocation  of  Funds 

The  first  research  question  addressed  by  this  study  pertained  to  the  use  of 
performance  assessment  and  performance  measures  in  the  allocation  of  state  funds  to 
postsecondary  education  institutions.  Was  performance  assessment  for  funding 
allocations  utilized  for  both  two-year  and  four-year  colleges  and  universities?  What 
portions  of  their  individual  budgets  did  the  states  allocate  in  this  manner?  Which  states 
had  used,  were  using,  and  planned  to  use  this  type  of  funding  allocation?  Did  states  use 
multiple  measures  of  institutional  performance  to  allocate  funds?  How  many  institutional 
performance  measures  did  states  utilize?  Were  institutional  performance  measures  the 
same  for  all  postsecondary  institutions?  Specifically,  were  there  different  measures  of 
performance  for  four-year  and  two-year  colleges  and  universities?  What  were  the 


measures  of  postsecondary  institutional  performance  most  frequently  used  for  the 
allocation  of  funds?  This  research  question  was  answered  through  a  review  of  the 
literature  and  was  addressed  in  chapter  two.  '        .  .  .  ^ 

Research  Question  Two 

Accuracy  and  Flexibility  of  Performance  Measures 

The  second  research  question  addressed  by  this  study  focused  on  the  degree  to 
which  the  institutional  performance  measures  utilized  by  states  and  institutions  accurately 
measured  the  performance  of  community  and  two-year  colleges.  Could  the  outcome  > 
measures  commonly  used  by  states  and  institutions  accurately  depict  community  and 
two-year  college  performance?  How  feasible  was  the  development  of  performance 
measures  to  account  for  differences  in  levels  and  types  of  institutions?  Specifically,  was 
there  a  model  that  could  be  used  as  an  accurate  performance  indicator  for  both  two-year 
and  four-year  postsecondary  institutions?  .  ,■■ ' 

This  research  question  was  answered  through  a  review  of  the  literature  and 
through  the  use  of  logistic  regression  analyses  to  develop  a  model  of  community  college 

performance.  This  research  question  is  addressed  in  chapters  two  and  three,  respectively. 

;■     ■  '      '  •. 

Evolution  and  Uses  for  Student  and  Institutional 
Performance  Measures 

Virtually  all  the  literature  concerning  student  and  institutional  performance 
assessment  confined  its  scope  to  four-year  residential  institutions— "It  ignore[d]  the  large 
and  growing  population  of  students  who  beg[an]  their  careers  in  postsecondary  education 
in  two-year  or  community  colleges."^  This  large  body  of  research  originally  focused  on 
the  affects  of  postsecondary  education  on  students'  lives.''  During  the  decade  of  the 
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1980s,  however,  performance  data  became  a  means  of  evaluating  and  motivating  efforts 
to  improve  undergraduate  education.^  More  recently,  student  and  institutional  outcome 
data  were  used  to  evaluate  the  productivity  and  performance  of  postsecondary  institutions 
and  systems.^  As  states  increasingly  used  institutional  performance  indicators  to  allocate 
funds  to  postsecondary  education  institutions — as  with  PBF&B — this  research  found  yet 
another  fimction. 

This  study  focused  specifically  on  the  community  college  system  for  four  distinct 
reasons,  which  follow.  ' 

Community  College  Enrollment  Size 

The  first  reason  for  focusing  on  the  community  college  sector  was  the  significant 
(and  often  overlooked)  role  of  these  institutions  in  the  nation's  postsecondary  education 
system.  Despite  the  fact  that  almost  half  of  the  beginning  postsecondary  education 
students  began  at  community  and  two-year  institutions,  the  majority  of  research  on 
college  student  success  and  institutional  performance  was  focused  on  the  traditional  four- 
year  college  sector.^  For  this  reason,  less  was  known  about  community  and  two-year 
colleges  and  the  students  who  attended  these  institutions  than  was  known  about  the  four- 
year  college  sector.'^ 

Traditional  Assumptions  about  College  Students 

A  second  reason  this  study  focused  on  community  colleges  was  because  most  of 
the  institutional  performance  measures  being  used  for  state  funding  allocations  were 
based  on  outdated  assumptions  about  postsecondary  student  populations. ' '  For  example, 
allocations  based  on  the  percentage  of  students  graduating  within  six  years  assumed  that 
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all  students  attended  college  on  more  than  a  part-time  basis,  that  they  all  intended  to 
obtain  a  degree,  and  that  all  entering  students  engaged  in  college-level  course  work 
during  the  first  term  of  college. 

Similarly,  fiinding  based  on  transfer  rates  to  four-year  colleges  assumed  that  all 
students  who  attended  community  colleges  intended  to  later  transfer  to  four-year 
colleges.'^  In  addition,  because  community  colleges  were  attended  by  a  larger  proportion 
of  part-time  students  than  were  traditional  four-year  colleges  and  universities,''' 
commonly  used  Pis  (such  as  time-to-degree)  did  not  adequately  capture  this  sector's 
contribution  to  higher  education. '  ^ 

The  "At-Risk"  Population 

The  third  reason  for  focusing  on  community  colleges  was  that  these  institutions 
had  the  largest  percentages  of  "at-risk"  students.  For  example,  according  to  the  National 
Center  for  Education  Statistics  (NCES),  freshmen  in  public  two-year  colleges  in  1995 
were  far  more  likely  to  enroll  in  remedial  courses  than  were  freshmen  in  public  four-year 
institutions  (41  versus  22  percent).'^  In  general,  the  "open-door"  admissions  policies  at 
community  colleges  meant  that  these  students  were  in  greater  need  of  academic 
counseling,  remedial  or  developmental  courses,  and  other  student  services  than  were 
students  attending  more  selective  four-year  colleges.'' 

Further,  although  the  "college  access  gap"  between  whites  and  blacks  and  whites 
and  Latinos  had  closed  fi-om  the  1 1-15  percent  range  to  5  percent  over  the  past  two 
decades,  research  showed  that  the  degree  completion  gap  remained  at  approximately  20 
percent  over  the  past  two  decades.'^  Since  two-year  colleges  were  more  likely  to  have 
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larger  percentages  of  minority  students  than  were  their  four-year  college  counterparts,  it 
followed  that  these  institutions  were  also  more  likely  to  experience  lower  graduation 
rates. 

Need  for  Different  Predictors  of  Success 

The  fourth  reason  for  focusing  on  community  colleges  was  because  past  studies 
on  student  predictors  of  success  had  dealt  primarily  with  traditional-age,  full-time  college 
freshmen  at  four-year  colleges  and  universities.^*^  Since  many  of  these  variables  were  not 
part  of  two-year  college  life,  some  of  the  predictors  of  success  for  four-year  college 
students  were  inapplicable  predictors  of  success  for  community  college  students.^' 

For  decades,  Alexander  Astin  and  Vincent  Tinto  studied  the  student 
characteristics  most  predictive  of  college  persistence  and  success  in  the  four-year  college 
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sector.    Their  work  supported  the  theory  that  pre-college  student  characteristics  were 
highly  predictive  of  student  success  in  college.  » 

More  recent  research  used  predictive  models  of  student  persistence  and 
graduation  as  a  means  of  assessing  the  performance  of  four-year  colleges.^^  Mortenson,^" 
along  with  Johnes  and  Taylor,^^  used  the  findings  of  Astin  and  Tinto  to  develop  a  means 
of  institutional  comparison  by  creating  models  that  took  into  account  those  variables 
largely  out  of  individual  institutional  control.^^  These  researchers  found  that  a  more 
accurate  assessment  and  comparison  of  four-year  institutional  performance  could  be 
obtained  by  accounting  for  various  student  characteristics  that  most  strongly  predicted  " 
persistence  and  graduation  rates  at  four-year  colleges.  This  method  could  also  be  applied 
to  the  two-year  college  sector,  assuming  the  longitudinal  data  was  available  for  students 
at  this  institutional  level. 


.    There  were  several  reasons  why  there  was  little  known  about  the  factors  that 
influenced  community  college  student  persistence  and  withdrawal  behavior.^^  One 
explanation  for  the  relative  lack  of  information  about  community  college  students  and 
institutions  was  the  difficulty  of  conducting  longitudinal  research  on  such  a  transient  - 
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cohort.    Adelman  found  that  "[t]he  increasing  complexity  of  [postsecondary  students'] 
attendance  patterns  [was]  one  of  the  most  significant  developments  in  higher  education  of 
our  time,  one  that  pose[ed]  grave  challenges  to  system-wide  planning  [and]  quality 
assurance."    The  tracking  of  community  college  students  tended  to  be  more  difficult 
than  tracking  traditional-age,  four-year  college  students,  since  community  college 
students  were  more  likely  to  transfer  many  times  before  completing  a  degree  or 
certificate  program.^*^  ^-  -   :       „•  ■         ■  : 

Data  Utilized 

By  utilizing  newly  available  data  about  community  college  students — ^the 

v' 

Beginning  Postsecondary  Students  Longitudinal  Study  (BPS:90/94)  conducted  by  the 
National  Center  for  Education  Statistics  (NCES)^  '—this  study  was  able  to  extend  the 
work  of  Astin  and  Tinto,  who  examined  four-year  college  student  predictors  of  academic 
achievement.^^ 

The  BPS:90/94  restricted-use  data  set  used  in  this  study  was  the  first  national 
longitudinal  study  to  include  a  large  and  comprehensive  number  of  variables  for  a 
nationally  representative  sample  of  community  college  students. "  Because  the 
BPS:90/94  included  both  traditional-  and  nontraditional-age  students,  this  sample  was 
representative  of  all  beginning  students  in  postsecondary  education.^"*  BPS:90/94 
provided  information  on  persistence,  progress,  and  attainment  of  students  from  their 
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initial  time  of  entry  into  postsecondary  education  through  their  leaving  school  and 
entering  the  work  force.^^ 

BPS: 90/94  followed  first-time,  beginning  students  for  five  years  (through  the 
spring  of  1994),  and  collected  student  data  and  financial  aid  reports.  By  starting  with  a 
cohort  that  had  already  entered  postsecondary  education  (from  the  NPSAS:90),^^  and 
following  it  for  five  years  (with  first  and  second  follow-ups  in  spring  of  1992  and  spring 
of  1994,  respectively),  BPS:90/94  determined  the  extent,  if  any,  to  which  students  who 
started  postsecondary  education  at  various  times  after  high  school  differed  in  their 
progress,  persistence,  and  attainment.^^  (Incidentally,  a  new  cohort  of  beginning  ; 
postsecondary  students  from  1995-96  was  followed  beginning  in  1998,  which  may 
provide  a  means  for  repeating  this  study  for  fiiture  generations  of  community  college 
students.) 

This  rich  data  source  from  NCES  allowed  for  the  development  of  a  predictive 
model  of  community  college  students'  graduation  and  transfer  behavior,  a  model  which 
could  later  be  applied  to  individual  institutional  data,  compared  to  actual  institutional 
graduation  and  transfer  behavior,  and  used  for  the  assessment  of  community  colleges. 
Because  the  BPS:90/94  restricted-use  data  set  allowed  this  researcher  access  to  individual 
student  data,  the  unit  of  analysis  was  the  student. 

Predictors  of  College  Student  Success 

Although  most  of  the  literature  on  institutional  assessment  focused  on  the  four- 
year  college  sector,  the  methodology^^  and  variables  used  for  this  sector  offered  a  logical 
starting  point  for  research  on  two-year  colleges  and  students.  Based  on  prior  research 
concerning  the  four-year  college  sector  and  its  students,""^  student-related  variables  were 
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selected  for  inclusion  in  a  multiple  logistic  regression  analysis.  The  variables  discussed 
below  were  analyzed  for  possible  use  in  the  regression  analysis. 

According  to  the  research  of  Jill  Johnes  and  Jim  Taylor"*',  entry  qualifications — 
entrance  exam  scores,  academic  ability,  and  type  of  high  school  attended — were  three  of 
the  strongest  student-related,  pre-coUege  variables  affecting  student  success  at  four-year 
colleges."*^  According  to  these  researchers,  other  strong  predictors  of  academic  success 
included  student  gender,  subject  mix  (specifically,  business,  social  science  or  language 
programs),  on-campus  residency,  and  starting  age  at  first  postsecondary  institution  (i.e., 
delayed  enrollment).'*^  * 

Astin's  research  supported  the  theory  that  student  characteristics  prior  to  entering 
postsecondary  education  were  the  most  important  predictors  of  college  success."*^ 
Specifically,  Astin  found  that  high  school  grades,  SAT  scores,  race,  and  sex  were  the 
strongest  predictors  of  college  student  persistence.'*^  Other  important  student  background 
characteristics  included  the  type  of  high  school  attended,  religion,  educational  level  of 
father  and  mother,  birth  order,  number  of  siblings,  family  income,  age,  high  school 
achievements,  average  grade  in  high  school,  and  the  achievement  of  twelve  specific 
honors  in  high  school.'*^ 

Astin's  research  also  indicated  that  a  student's  fiiture  plans— probable  field  of 
study  in  college,  probable  career  choice,  highest  degree  sought,  expectations  about 
getting  married  during  or  after  college,  and  concern  about  finances— had  only  modest 
predictive  power  for  college  achievement.'*^ 

Overall,  Astin's  research  provided  evidence  that  four  student  characteristics 
improved  the  predictive  validity  of  freshmen  grade  point  average  (GPA)  when  sex,  high 
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school  grades,  SAT/ACT,  and  college  selectivity  were  controlled:  (1)  membership  in  a 
scholastic  honor  society;  (2)  attendance  at  a  private  high  school;  (3)  high  self-rating  on 
academic  ability;  and  (4)  high  self-rating  on  drive  to  achieve."** 

Astin's  work  also  focused  on  the  prediction  of  drop-out  behavior  in  college 
students.     He  found  that  the  males  who  (1)  were  older  than  average,  (2)  were  seeking 
less  than  a  bachelor's  degree  on  admission,  (3)  reported  "some  chance"  of  getting 
married  while  in  college,  (4)  attended  public  high  school,  (5)  majored  in  physical  science, 
(6)  were  a  religion  other  than  Protestant,  (7)  frequently  overslept  and  missed  classes  or 
appointments,  and  (8)  turned  in  papers  late,  were  more  likely  to  drop  out  of  college.^*^ 

Astin  found  many  of  the  same  variables  to  be  predictive  of  female  drop-out 
behavior.^'  By  contrast,  however,  the  females  most  likely  to  drop  out  of  college  were 
those  who  had  major  concerns  about  finances  on  admission  to  college,  were  non-white, 
had  no  religious  preference,  and  did  not  win  a  varsity  letter  in  high  school." 

According  to  the  U.S.  Department  of  Education,  the  "higher  a  students' 
socioeconomic  status  (SES),  the  more  likely  that  student  was  to  transfer  to  a  four-year 
institution:  35  percent  of  high  SES  students  transferred  compared  to  21  percent  of  middle 
SES  and  7  percent  of  low  SES  students."^^ 

Tinto's  research  also  indicated  that  "individuals  of  different  race,  ability,  and 
social  status  origins  differed  markedly  in  the  rate  at  which  they  left  higher  education 
within  four  years  of  entry  without  earning  a  college  degree."^^  Tinto  used  variables  such 
as  student  "intention"  and  "commitment"  as  predictors  of  student  success  because  he 
considered  these  to  be  the  "primary  roots"  of  departure  from  higher  education.^^  Tinto 
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also  found  that  individual  attributes  such  as  sex,  age,  ethnicity,  expected  major,  and 
marital  status,  had  predictive  validity.^^ 

Tinto's  earlier  work  focused  on  what  he  termed  academic  and  social 
integration.^^  A  student's  average  undergraduate  grades  and  membership  in  a  scholastic 
honor  society  measured  academic  integration,  which  Tinto  found  to  be  highly  predictive 
of  student  success.^^  Social  integration,  a  student's  involvement  with  peers  and  faculty, 
was  defined  by  how  well  the  student  knew  a  professor  or  administrator,  whether  the 
student  was  president  of  one  or  more  student  organizations,  had  a  major  part  in  a  play, 
won  a  varsity  letter,  or  edited  a  school  publication.^^ 

Definition  of  Terms 

A  performance  indicator,  for  the  purposes  of  this  paper,  was  "a  numerical  value 
which  provid[ed]  a  measurement  for  assessing  the  quantitative  performance  of  a 
system."^" 

Performance  budgeting  and  performance  funding,  though  similar  in  method 
and  implication,  had  several  distinctions.  According  to  the  National  Council  for 
Research  and  Planning  (NCRP),  performance  budgeting  was:  "the  subjective  practice 
that  [took]  into  account  various  areas  of  performance  in  determining  an  institution's 
overall  appropriation  and  [let]  state  officials  retain  discretion  over  how  much  to  alter  the 
college's  bottom  line."^' 

Performance  budgeting,  according  to  Burke,  was  when  "state  governments  or 
coordinating  boards  consider[ed]  institutional  performance,  usually  based  on  a  longer  list 
of  indicators,  as  a  general  context  when  determining  the  total  budgets  of  public  colleges 
and  universities."''  Burke  and  Serban  added:  "performance  budgeting  considers  results 
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of  performance  indicators  as  a  factor  in  the  total  funding  of  public  colleges  and 
universities."^^  With  performance  budgeting,  there  was  a  flexible  and  discretionary  but 
unclear  link  between  performance  indicators  and  allocated  funding.^ 

Performance  funding,  according  to  the  National  Council  for  Research  and 
Planning  (NCRP),  "use[d]  a  formula  to  determine  how  much  to  give  a  college  that  ha[d] 
met  a  prescribed  goal  or  specified  standards  of  performance."^^  Burke  and  Serban  put  it 
another  way:  "performance  funding  ties  special  sums  directly  to  results  of  specific 
indicators."^^ 

Outcomes  measures  were  such  things  as  graduation  rates,  transfer  rates,  mean 
income  of  alumni  five  years  after  graduation,  student  satisfaction  measures,  student  test 
scores,  and  other  variables  focusing  on  the  end  product  of  a  student's  educational 
experience. 

Institutional  graduation  rates,  for  this  study,  equaled  the  percentage  of  students 
completing  a  degree  or  certificate  program  within  a  specific  period  of  time. 

Transfer  rates  required  the  accurate  tracking  of  students  Irom  one  institution  to 
another.^^  According  to  the  U.S.  Department  of  Education,  transfer  "[did]  not  imply 
transfer  of  credit ...  it  simply  characterize [d]  observed  transitions  between  institutions. 
For  transfer  to  a  less-than-four-year  institution,  any  transition  that  was  not  followed  by  a 
return  to  the  first  institution  constitute  [d]  a  transfer.  For  transfer  to  a  four-year 
institution,  any  entry  into  a  four-year  institution  constitute [ed]  a  transfer,  regardless  of 
subsequent  enrollment  at  the  institution  of  origin."^*  In  addition,  "students  were  only 
classified  as  transfers  if  they  changed  institutions  permanently  without  returning  to  the 
original  institution."^^  This  definition  excluded  students  who  "were  enrolled  at  two 
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institutions  simultaneously,  attended  summer  school  at  a  different  institution,  or  were 
enrolled  temporarily  at  a  second  institution  and  then  returned  to  the  first  one."^° 

The  residual,  as  used  in  this  study,  meant  the  difference  between  the  predicted 
graduation  and/or  transfer  rate  at  an  institution  and  that  institution's  actual  graduation 
and/or  transfer  rate.  .    •;     ?    -  . .  • 

Predictors,  as  used  in  this  study,  were  variables  that  correlated  with  the 
graduation  and/or  transfer  behavior  of  postsecondary  college  students  five  years  after 
beginning  their  college  careers. 
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CHAPTER  2 
POSTSECONDARY  EDUCATION  FUNDING 

Community  College  Funding 

History  of  Community  Colleges 

Since  the  inception  of  community  colleges  in  1901,  these  institutions  have  been 
highly  adaptive  to  changes  exerted  by  external  forces.'  The  enormous  growth  of  these 
institutions  during  the  twenty-five  years  following  World  War  II  resulted  in  a  tripling  of 
the  number  of  community  colleges  to  nearly  thirteen  hundred  institutions.^  Campbell 
reported  that  community  colleges  of  the  decade  of  the  1990s,  initially  extensions  of  the 
public  high  school  system,  ranged  "from  comprehensive  community  college  districts  and 
systems  to  two-year  campuses  of  universities  to  vocational-technical  colleges,"  and 
enrolled  half  of  all  beginning  college  students.  ^ 

Two  landmark  reports  marked  the  beginning  and  the  end  of  the  greatest  growth 
period  in  community  college  history.  The  1947  report  of  President  Truman's 
Commission  on  Higher  Education  advocated  free  schooling  through  grade  fourteen  for  all 
who  could  benefit.  ^  At  that  time  there  were  only  3 1 5  public  junior  colleges,  and  only  12 
percent  of  individuals  between  the  ages  of  eighteen  and  twenty-four  were  enrolled  in 
college  (four-  or  two-year).^  Twenty-three  years  later,  when  the  Carnegie  Commission 
on  Higher  Education  published  The  Open-Door  College,^  there  were  847  public 
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community  colleges,  and  the  proportion  of  eighteen-  to  twenty-four-year-olds  enrolled  in 
college  had  risen  to  32  percent/ 

Sources  of  Financial  Support  for  Community  Colleges 

Change  and  uncertainty  have  marked  the  financial  support  for  community 
colleges  throughout  their  history.  As  "the  largest  area  of  state  funding  that  [was]  not 
constitutionally  mandated,"^  higher  education  fiinding  was  known  as  the  "budget 
balancer."^  In  the  face  of  competing  social  program  needs,  higher  education  was  the 
largest  component  in  state  budgets  that  could  be  "readily  cut  or  held  to  small  increases."" 

According  to  Campbell,  Leverty,  and  Sayles,  funds  for  higher  education  in  the 
decade  of  the  1990s  came  from  five  major  sources:  the  federal  government,  state 
government,  local  government,  tuition  and  fees  (combined),  and  other  sources."  In 
1996,  for  example,  Cohen  and  Brawer  reported  that  the  distribution  of  funds  for  higher 
education  came  from  five  major  funding  sources:  46  percent  from  state  government,  1 8 
percent  from  local  government,  20  percent  from  tuition  and  fees,  5  percent  from  the 
federal  government,  and  3  percent  from  other  sources.     Clearly,  state  funding  was  the 
largest  source  of  revenue  for  postsecondary  education. 

Postsecondary  education's  heavy  reliance  on  state  funding  made  it  particularly 
susceptible  to  changing  priorities  at  this  level  of  government.'^  For  example,  colleges 
had  to  respond  to  the  overall  decrease  in  real  state  support  for  community  colleges  that 
occurred  during  the  1990-1995  period."*  Decreases  in  state  support  for  community 
colleges  fiieled  a  trend  toward  increased  reliance  on  external  revenue  sources — such  as 
government  grants  and  contracts — and  away  from  core  state  and  local  fionding  for  basic 
college  operations. '  ^ 
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Performance  Assessment  for  the 
Allocation  of  Funds  to  Community  Colleges 

Input-Based  Funding  as  Precursor 

According  to  Burke,  public  colleges  and  universities  were  traditionally  fiinded 
through  formulas  that  relied  on  "current  costs,  student  enrollments,  and  inflationary 
increases."'^  Funding  for  community  colleges  often  depended  on  enrollment  figures — 
such  as  full-time  equivalent  student  measurements  (PTEs) — to  dictate  the  amount  of  state 
funds  allocated  per  institution.  Because  they  focused  on  variables  that  were,  for  the  most 
part,  out  of  the  control  of  the  institution.  Burke  called  these  funding  approaches  "input" 
methods  of  distribution.'^ 

The  use  of  "input  funding"'^  was  particularly  popular  and  relatively  effective 
during  postsecondary  education's  high  growth  periods.'^  One  such  period  was  the  post- 
World  War  II  era  during  which  the  U.S.  established  the  G.I.  Bill,^°  allowing  displaced 
war  veterans  to  return  to  college  free  of  charge.  The  resulting  increase  in  college  • 
enrollments — the  largest  in  U.S.  history — encouraged  policymakers  to  utilize  allocations 
based  on  inputs  such  as  enrollment-based  funding.^'  '■ 

In  a  less  affluent  era  shortly  after  the  1960s,  enrollment-driven  fiinding  formulas 
became  less  usefiil  and  less  prevalent.^^  With  the  slowing  of  growth  in  higher  education, 
input-driven  formulas  met  with  criticism  fi-om  legislative  bodies  and  the  public.^^  Palmer 
attributed  this  criticism  of  input-driven  formulas  to  the  fear  that  this  ftmding  method 
encouraged  an  "enrollment-at-all-costs  attitude,"^''  often  detrimental  to  an  institution's 
academic  viability.^^  Burke  echoed  the  common  criticism  that  input-driven  formulas 
focused  attention  on  raw  numbers  of  enrollees  while  ignoring  the  length  of  time  students 


attended  an  institution.    According  to  Burke,  input-driven  funding  "neglect[ed]  outputs 
and  outcomes,  such  as  the  quantity  and  quality  of  graduates  and  the  range  and  level  of 
research  and  service  delivered  to  states  and  society."^'' 

,    According  to  Campbell,  Leverty,  and  Sayles,  input-driven  funding  formulas  posed 
a  particular  problem  for  community  colleges  because  "significant  portions  of  the  mission 
of  these  institutions  [were]  not  associated  with  enrollment  or  credit  hours."^*  The 
community  college  mission  often  emphasized  continuing  education  and  public 
service — areas  that  were  usually  sustained  by  self-supporting  or  fee-for-service  ftmding?' 

The  Accountability  Era 

The  decade  of  the  1 990s  saw  a  shift  from  enrollment-driven  "input"  formulas 
toward  models  that  focused  on  an  institution's  demonstrated  outcomes.^*^  According  to 
Campbell,  Leverty,  and  Sayles,  as  resources  declined  and  competition  for  funds  increased 
in  the  decade  of  the  1990s,  state  legislatures  began  imposing  accountability  on  publicly 
supported  institutions.^' 

Prior  to  the  decade  of  the  1990s,  "most  ftmding  systems  [for  higher  education] 
[did]  not  consider  state  educational  goals,  nor  [did]  they  provide  incentives  for 
outcomes."^^  In  the  decade  of  the  1990s,  state  leaders  began  to  view  traditional  forms  of 
higher  education  ftinding  as  "obsolete, . . .  insulated  from  state  and  institutional  strategic 
planning,  unresponsive  to  state  and  national  priorities,  and  inappropriately  separated  from 
the  question  of  value."^'' 

Reasons  for  Performance-Based  Funding  and  Performance-Based  Budgeting 

According  to  the  National  Council  for  Research  and  Planning  (NCRP) 
"performance-based  ftinding  and  budgeting  represent[ed]  a  dramatic  departure  from 
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traditional  budgeting."^''  Changes  in  higher  education  funding  followed  an  increased 
public  call  for  accountability  that  moved  postsecondary  education  to  a  system  whereby 
concrete  educational  outcomes  were  defined,  measured  and  documented.^^  In  some 
respects,  performance-based  budgeting  (PBB)  and  performance-based  funding  (PBF) 
were  responses  to  a  question  posed  by  the  Knight  Higher  Education  Collaborative:  "What 
would  happen  if  an  institution  [was]  reimbursed  for  its  costs  only  to  the  degree  that  the 
student  had  demonstrated  mastery  of  the  subject?"^^ 

"The  idea  of  accountability  [was] ...  the  expression  of  a  wish  felt  by  many  public 
officials,  donors,  and  the  general  public  as  well — that  the  increasing  sums  laid  out  for 
public  services  ought  to  be  justified  by  reliable  estimates  of  the  outcomes."^^  The 
"emphasis  on  outcomes  . . .  reflect[ed]  taxpayers'  desire  to  know  their  money  [was]  being 
well  spent."'* 

While  the  demand  for  higher  education  continued  to  outpace  public  revenues,'^ 
and  America's  trust  in  higher  education  institutions  continued  to  wane,'*^  policymakers 
struggled  to  find  a  "logical  basis  for  higher  education  funding.""'  According  to  Albright, 
"performance-based  fianding  represent[ed]  a  paradigm  shift.  Rather  than  a  state  meeting 
the  institution's  needs,  the  college  or  university  me[t]  the  state's  needs.""^  .  - 

The  literature  presented  five  broad  reasons  for  the  advent  of  performance-based 
fiinding  and  budgeting  (PBF&B):  (1)  a  general  dissatisfaction  by  the  public  and  state  and 
federal  legislators  with  the  quality  of  higher  education;    (2)  the  alarming  growth  of     . , 
demands  by  higher  education  institutions  for  financial  resources  relative  to  enrollment;"^ 
(3)  growing  student  financial  burdens  in  the  form  of  rising  tuition  costs;"*^  (4)  the 
emergence  of  a  financially  sawy  public  that  demanded  returns  on  its  investment  in 
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education;''^  and  (5)  trends  in  business-sector  management  practices  toward  the  use  of 
performance  indicators  (Pis)  to  measure  performance.'*^  Meritosis  and  Wolanin  added 
that  "shifting  government  policies  and  priorities,  growing  demands  from  consumers,  and 
an  evolving  relationship  with  employers  who  rel[ied]  on  community  colleges  to  provide 
them  with  skilled  workers"  also  contributed  to  the  move  toward  PBF/^ 
Waning  public  conndence 

According  to  Bowen,  the  accountability  era  "reflect[ed]  in  part  a  failure  of 
[public]  confidence  in  many  of  our  institutions."'*^  Serban  agreed  that  "performance 
funding  for  public  higher  education  . . .  emerged  in  response  to  ...  the  public's  concern 
for  accountability  and  results."^° 

Criticisms  of  two-year  and  community  college  performance  also  served  as  an 
impetus  for  a  move  to  accountability  practices."'*'  Bray  and  Kanter  reported  that  outcomes 
assessment  efforts,  achieved  through  grassroots  networks  during  the  early  1980s,  were  a 
"response  to  the  calls  of  critics  who  claimed  community  colleges  were  a  'revolving  door' 
for  students.""  In  California  such  claims  resulted  in  the  formation  of  an  eighty-college 
collaboration  known  as  the  Learning  Assessment  Retention  Consortium  (LARC)  in 
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1981.    According  to  the  Knight  Higher  Education  Collaborative,  "the  sad  truth  [was] 
that  too  many  who  cross[ed]  the  starting  line  never  reach[ed]  the  finish— whether  defined 
as  the  completion  of  a  baccalaureate  degree,  an  associate's  degree,  or  a  basic  skills 
program.  "^"^ 

Nichols  reported,  "[T]he  purposes  of  two-year  colleges  [were]  often  misconstrued 
by  the  public  and  by  other  segments  of  postsecondary  education."^^  According  to  Welleri 
and  Seybert,  "Concern  [was]  raised  about  the  success  rate  of  students  entering 
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community  colleges,  as  well  as  the  ability  of  the  colleges  to  move  students  on  to  the 
bachelor's  degree."^^ 

General  perceptions  about  the  value  and  quality  of  higher  education  also  played  a 
major  role  in  the  move  toward  accountability  funding."  For  example,  Austin  R.  Gilbert, 
as  chairman  of  South  Carolina's  Commission  on  Higher  Education,  reported  in  1997  that 
his  constituents  doubted  that  public  colleges  were  using  their  tax  dollars  wisely.^* 
According  to  Gilbert,  higher  education  institutions  were  "faced  with  a  crisis  in 
credibility."^' 

Drawing  from  a  1989  poll  conducted  to  assess  the  quality  of  postsecondary 
education,  Langfitt  reported  that  a  significant  majority  of  respondents  felt  that  the  quality 
of  higher  education  had  declined  over  the  past  1 0  to  1 5  years.^°  This  sentiment  was 
echoed  throughout  the  literature:  "[A]  combination  of  inefficiency  and  perceived 
insensitivity  on  the  part  of  the  institutions  toward  their  clients  [was]  at  the  heart  of  the 
public's  disaffection  with  . . .  higher  education."*' 
Rising  cost  of  tuition 

"Financing  a  college  education  [was]  a  serious  and  troublesome  matter  for  the 
American  people."*^  Langfitt  reported  that  the  most  common  criticisms  aimed  at  higher 
education  were  that  "colleges  and  universities  [got]  all  they  [could]  and  spen[t]  all  they 
[got],  that  teaching  assistants  provide[d]  much  of  the  undergraduate  education  . . .  [and 
that  a]  college  education  [was]  not  worth  its  cost.""  The  National  Committee  on  the 
Cost  of  Higher  Education  reported  that  "rising  college  tuitions  [were]  real.  In  the  twenty 
years  between  1976  and  1996,  the  average  tuition  at  public  universities  increased  from 
$642  to  $3,151."^ 
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"[T]here  [was]  ample  public  criticism  reflected  in  the  media,  letters  from  parents 
and  alumni,  and  in  occasional  student  strikes  about  rising  costs  of  higher  education."^^ 
According  to  Langfitt,  even  as  early  as  1976 — prior  to  the  push  for  PBF«&:B  in 
America — the  call  for  accountability  reflected  the  public's  frustration  over  rapidly 
increasing  costs  at  higher  education  institutions.^^  Bowen  concurred  that  "the  call  for 
accountability  . . .  reflect[ed]  in  part . . .  fiiistration  over  rapidly  increasing  costs."^' 

Johnstone  responded  to  this  concern  and  its  subsequent  criticisms  by  pointing  out 
that  "tuitions  [rose]  faster  than  the  rate  of  inflation."^*  He  concluded  that  the  general 
public  and  the  media  could  not  be  expected  to  understand  the  fact  that  the  Consumer 
Price  Index  (CPI),  the  gauge  by  which  inflation  was  measured,  was  an  average  of  price 
increases  across  goods.  According  to  Johnstone,  by  virtue  of  being  an  average,  one  half 
of  the  goods  included  in  this  index  exceeded  the  CPI.  He  reported  that  "tuition  [would] 
almost  always  be  in  that  above-average  half."^^ 

Langfitt  analogized  the  criticisms  waged  at  higher  education's  rising  costs  to  the 
public  perception  of  the  health  care  industry  in  the  decade  of  the  1980s.  Langfitt 
described  college  and  university  finance  as  a  "cost-plus  system"  whereby  colleges  and 
universities  "calculate[d] . . .  costs,  then  [sought]  the  revenues  to  cover  the  costs,  often 
employing  tuition  and  fee  increases  to  balance  the  budget."'" 

According  to  Langfitt,  public  health  care  and  higher  education  played  similar 
roles  in  developing  and  maintaining  the  human  capital  "that  [was]  the  source  of  national 
progress  and  prosperity."' '  Langfitt  reported  that  both  institutions  experienced  the  same 
public  disenchantment— "excessive  costs,  a  perceived  decline  in  the  quality  of  the 
services,  and  inadequate  access  based  mainly  on  costs."  Langfitt  added  that  the  public's 
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"dissatisfaction  with  colleges  and  universities  . . .  [was]  on  the  rise — mainly  focused  on 
cost  increases  in  tuition  and  fees  considerably  in  excess  of  general  inflation."'^ 
Business-oriented  models  as  precursors 

PBF&B  also  resulted  from  the  evolution  of  resource  allocation  and  management 
methods  in  the  private  business  sector^^  According  to  Alfred  and  Carter,  emerging 
community  college  management  practices  were  "similar  to  management  iimovations  in 
for-profit  organizations"/'' 

Process-oriented  evaluation  methods,  such  as  Continuous  Process  Improvement 
(CPI)  and  Total  Quality  Management  (TQM),  contributed  to  the  development  of 
performance-funding  methods/^  The  influences  of  CPI  and  TQM  were  most  prevalent  in 
Pis  that  focused  on  the  "improvement  of  services  to  students."^^  Research  conducted  by 
Dooris  and  Teeter,  for  example,  illustrated  that  Pis  could  be  effectively  derived  from 
quality  improvement  methods  such  as  TQM  and  CPlJ^ 

Postsecondary  administrators  and  faculty  challenged  the  applicability  of  CPI  and 
TQM  to  higher  education,  because  these  business-oriented  methods  employed  language 
unfamiliar  to  and  often  conceptually  inappropriate  for  the  education  sector/^  Borden  and 
Banta  pointed  out  that  the  use  of  the  phrase  "'customer  satisfaction'  [was]  not  generally 
accepted  as  an  appropriate  focus  for  classroom  and  curricular  development,"'''  despite  the 
fact  that  "most  legislators  [said]  they  want[ed]  higher  education  to  operate  more  like  a 
business."^'' 

Call  from  legislative  and  budgetary  bodies  and  decreased  tax  support 

Other  factors  that  contributed  to  the  call  for  accountability  in  higher  education 
included  declining  resources,  increased  competition  for  fiinding  (especially  in  those  states 
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experiencing  rapid  growth  in  their  school-  and  college-age  populations),  and  an  overall 
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paradigm  shift  among  legislative  bodies.    For  these  reasons  "[IJawmakers  [were] 
reluctant  to  spend  more  on  higher  education."^^  This  hesitation  resulted  in  a  dwindling  of 
the  funds  once  earmarked  for  higher  education.^^ 

Langfitt  reported  in  1 986  that  federal  government  contributions  to  higher 
education  revenues  were  down  2  percent  from  1976  levels.*'*  In  California  the  "declining 
resources  caused  by  state  response  from  the  legislature  and  the  California  Community 
Colleges  Board  of  Governors"  were  a  key  reason  for  the  "vigorous  oversight  of  fiinds 
allocated  to  community  colleges."*^  California  used  assessment  of  outcomes  as  a  tool  for 
distributing  scarce  state  resources  to  those  institutions  that  "demonstrate [d]  educational 
quality,  efficiency,  and  effectiveness."*^ 

Borden  and  Banta  also  recognized  increasing  demands  for  public  fiinds.*' 
According  to  these  researchers,  competition  for  state  fiinds  grew  dramatically  in  the 
decade  of  the  1990s,  and,  as  a  result,  "higher  education  saw  its  support  diminish  as  health 
care,  care  for  the  aged,  and  penal  system  demands  increased."**  Increased  demand  for 
financial  resources  relative  to  enrollments  caused  some  states  to  move  toward  student 
outcomes  assessment  programs.*^  In  the  United  Kingdom  and  other  European  countries, 
the  development  of  higher  education  Pis  was  due  almost  completely  to  political  pressures 
for  fiinding  priorities  and  accountability.^ 

There  was  also  a  general  shift  in  the  way  legislators  thought  about  public  higher 
education.^'  Despite  the  possible  cause  of  policymakers'  concerns,  there  was  a  "clear 
need  for  academics  to  address  decision  makers'  needs  for  accountability  data  if  the 
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provision  of  resources  to  support  those  improved  programs  [was]  to  continue  in  the 
future."^^ 

A  more  sophisticated  public 

A  greater  societal  knowledge  of  economics  and  finance  was  another  source  of  the 
increased  demand  for  performance  assessment  of  institutions:  "[The  general  public]  was 
more  knowledgeable  about  national  and  global  economic  issues  due  to  intensive  media 
coverage  . . .  and  manage[d]  their  personal  financial  affairs  with  more  care."^^  In  return, 
society  was  placing  ever-greater  demands  on  higher  education  to  adequately  prepare 
students  for  the  modem  workplace.^'*  Ball  and  Wilkinson  reported  that  higher  education 
was  simply  part  of  a  greater  move  towards  a  "performance  oriented  culture" — a  "society 
striving  for  'value  for  money. '"^^ 

Other  groups,  such  as  the  National  Council  for  Research  and  Planning  (NCRP), 
reported  that  the  availability  of  performance  outcomes  information  and  the  need  for  better 
and  more  flexible  service  delivery  were  other  factors  contributing  to  the  move  toward 
PBF&B  for  higher  education.^^  '  ' 

In  the  end,  the  call  for  accountability  necessitated  a  "definition  of  costs  as  a 
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fimction  of  what  [was]  done  and  how  it  [was]  done,"  and  resulted  in  a  "process  of 
consensus  building  among  the  primary  constituencies:  policy  makers,  administrators, 
faculty,  students  and  the  public."^^  This  consensus-building,  as  Brown  pointed  out,  was 
"often  difficult  because  each  evaluator  approache[d]  the  university  with  his  own  special 
concerns,  his  own  consfituencies,  his  own  prejudices,  and  his  own  objectives. "^^ 
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Status  of  Performance  Funding  and 
Performance  Budgeting  in  the  Late  1990s 

In  1979,  Tennessee  was  the  first  state  to  adopt  a  performance  funding  program  to 
assess  the  progress  of  publicly  funded  higher  education.^^  With  the  exception  of 
Tennessee,  the  states  that  originally  used  performance  funding  began  their  programs  in 
the  first  half  of  the  decade  of  the  1990s.  "^"^  By  1997,  Arkansas,'"'  Colorado,"^^  Florida,'*^- 
Kentucky,'"^  Minnesota,"'^  Missouri,"'^  Ohio,"'^  South  Carolina, "'^  and  Tennessee"^ 
were  using  performance-based  funding  or  budgeting  methods. "° 

Incongruity  in  the  Literature 

The  literature  on  state  participation  in  performance  fianding  and  budgeting  was 
inconclusive.  Two  possible  reasons  for  the  literature's  incongruity  were  the  lack  of 
standardized  definitions  for  performance  fianding  and  budgeting,  and  the  erratic  nature  of 
state  participation  in  this  form  of  funding. 

Because  the  definitions  of  PBF&B  varied  slightly  throughout  the  literature, 
researchers  drew  differing  conclusions  about  state  participation  in  this  type  of  funding. 
The  NCRP,  for  example,  defined  performance  budgeting  as  "the  subjective  practice  that 
[took]  into  account  various  areas  of  performance  in  determining  an  institution's  overall 
appropriation  and  [let]  state  officials  retain  discretion  over  how  much  to  alter  the 
college's  bottom  line."' ' '  On  the  other  hand,  the  NCRP  defined  performance  funding  as 
a  system  that  "use[d]  a  formula  to  determine  how  much  to  give  a  college  that . . .  met  a 
prescribed  goal  or  specified  standard  of  performance.""^  Because  the  differences 
between  performance  fimding  and  budgeting  were  often  misunderstood,  reporting  errors 
occurred  at  the  state  and  institutional  level. 
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The  incongruity  in  the  Hterature  about  state  participation  in  performance  funding 
and  budgeting  was  also  due  to  states'  ever-changing  participation  in  these  programs.  For 
example,  a  survey  conducted  by  the  Rockefeller  Institute  of  Government  in  1997 
indicated  that  ten  states  used  performance  funding;  eight  were  likely  to  continue  using 
this  funding  method;  and  seventeen  states,  along  with  the  District  of  Columbia,  were 
"likely  to  adopt  it  over  the  following  five  years."  '    Serban  reported  that  eleven  states 
used  performance  funding  in  1998."^  Only  four  months  later,  the  NCRP  reported, 
"[0]ver  half  of  the  states  base[d]  some  of  their  allocations  on  the  campuses' 
performance,"  and  "all  but  five  states  indicate[d]  they  [were]  likely  to  do  so  within  five 
years."" ^  In  another  report  from  July  of  1998,  the  State  Higher  Education  Executive 
Officers  (SHEEO)  reported  that  "over  40  percent  of  the  states  use[d]  performance 
measures  in  the  budgetary  process  in  some  manner."' ' ^ 

In  October  1998,  the  NCRP  reported  that  fourteen  states  used  PBF,  as  defined  by 
that  organization."^  At  that  time,  twenty-one  states  used  performance  budgeting,  which 
"[took]  into  account  various  areas  of  performance  in  determining  an  institution's  overall 
appropriation  and  let  state  officials  retain  discretion  over  how  much  to  alter  the  college's 
bottom  line.""^ 

Table  2-1  and  Table  2-2  list  the  states  that  were  likely  to  adopt,  had  already 
adopted,  and  were  likely  to  continue  performance  funding  and  performance  budgeting  in 
early  1998,  according  to  Burke  and  Serban. Arkansas's  discontinuance  of  its  program 
was  due  in  part  to  "a  perception  that  two-year  colleges  did  not  receive  their  fair  share  of 
total  state  funding  for  higher  education."'^'  Burke  and  Serban  reported  that  Tennessee'^^ 
and  Missouri, '^^  among  others,  appeared  to  have  committed  themselves  to  this  practice 
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for  the  long  term.  According  to  Albright's  1998  report,  both  Arkansas  and 
Kentucky '^^  abolished  their  programs  in  1997.'^^  . 

Portion  of  Budget  Allocated  through  PBF  and  PBB 

The  amount  of  money  appropriated  for  PBF&B  to  public  colleges  varied  from 
state  to  state.  South  Carolina,  for  example,  signed  Act  359  into  law  in  the  spring  of  1996, 
which  made  it  the  first  and  only  state  to  plan  to  base  all  of  its  appropriations  for  higher 
education  entirely  on  how  well  an  institution  performed.     South  Carolina's  plan 
included  allocating  its  entire  budget  for  higher  education  based  on  performance  by  the 
year  2000.'^^ 


Table  2-1 .  Performance  Funding  for  Public  Higher  Education  in  1998. 


States  Likely  to  Adopt 
Performance  Funding 

States  That  Have  Adopted 
Performance  Funding 

States  Likely  to  Continue 
Performance  Funding 

Alaska 

Colorado 

Colorado 

District  of  Columbia 

Connecticut 

Florida 

Illinois 

Florida 

Minnesota 

Indiana 

Kentucky 

Missouri 

Louisiana 

Minnesota      .  v  '  ■ 

Ohio 

Maine 

Missouri 

South  Carolina 

Maryland 

Ohio 

Tennessee 

Massachusetts 

South  Carolina 

Washington 

Nebraska 

Tennessee 

Nevada 

Washington 

New  York 

North  Dakota 

Oregon 

South  Dakota 

Texas 

Utah 

West  Virginia 

Wyoming 

Source:  Joseph  C.  Burke  and  Andreea  M.  Serban,  "Performance  Funding:  Fad  or  Emerging 
Trend?"  Community  College  Journal  (December/January  1997-1998):  28. 
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Table  2-2.  Performance  Budgeting  for  Public  Higher  Education  in  1998. 


States  Likelv  to  Adnnt 
Performance  Budgeting 

States  That  Have  Adonted 
Performance  Budgeting 

States  Ijkdv  to  C*oiitiniie 
Performance  Budgeting 

rxlaVcLllla 

/\laDania 

A  n  Trvnci 
r\.l  1Z.U1ICI 

r  lUI  lUa 

r  lUnUd 

Oi^itrirt  of  r^nliimhifl 

14q  Wyii  i 

1  la  Well  1 

T-4f)\\/$ii  1 
I  law  all 

VJCL/I  ^la 

Idsho 

Iddho 

IIIUICIIICI 

IvltUUC  lalallU 

L/UUlalalld. 

i>eurasKa 

Texas 

ivialllc 

Knoae  isianu 

IVI QJI P  h  1 1 C  PttQ 
I VI  a^^aw  11  Ud  wild 

1  CAaa 

\^  1  n  n  p  c  of  51 

IVllllllWOUld 

Montana 
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Most  states,  however,  tied  only  small  portions  of  their  budgets  to  performance 
measures.  Schmidt  reported  that  it  was  usually  less  than  ten  percent  of  the  budget. '^^ 
Although  the  funds  may  have  been  only  a  small  portion  of  the  total  budget,  the 
discretionary  nature  of  these  funds  made  them  increasingly  attractive  and  a  strong 
financial  incentive  for  improvement. '""^  Burke  and  Serban  agreed  that  small  sums  of 
money  were  just  as  effective  as  large  sums  in  creating  change  in  higher  education. 
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Measures  of  Institutional  Performance 


The  success  of  PBF&B  relied  heavily  on  the  development  of  accurate  Pis  to 
allocate  the  level  of  funding  an  institution  received.'''^  Without  the  "adoption  and  pursuit 
of  appropriate  indicators,"  decisions  might  have  been  made  which  "conflict[ed]  with 
institutional  policies."''*''  Pis  came  in  many  different  forms,  were  developed  by  many 
different  constituencies,  and  were  used  in  many  different  ways.  Variety  was  the  key  to 
capturing  the  diversity  of  the  nation's  postsecondary  institutions. 

Definition  of  Performance  Indicator 

Much  of  the  literature  on  Pis  failed  to  address  the  actual  definition  of  the  term.'''' 
In  1987,  Cuenin  defined  Pis  as  "numerical  values  which  provide[d]  a  measurement  for 
assessing  the  quantitative  performance  of  a  system.""*^  In  response  to  the  creation  of 
inaccurate  Pis,  Elton  wrote  cynically:  "Whatever  is  easily  measurable  becomes  a  PI."'''* 

Three  Types  of  Performance  Assessment 

Outcomes  assessment,  although  an  important  part  of  PBF  and  PBB,  was  not  the 
only  type  of  assessment  conducted  at  most  institutions.'"^  Pis  came  in  three  basic  forms: 
input,  process,  and  output  indicators. ' '° 
Input  indicators 

Input  indicators  focused  on  such  things  as  student  enrollment,  average  SAT  score 
of  entering  fi-eshmen,  and  number  of  books  in  the  library.'^'  Heavily  relied  upon  during 
postsecondary  education's  high  growth  period  from  the  late  1950s  to  the  decade  of  the 
1970s,  input  indicators  reflected  the  first  component  of  the  institutional  performance 
equation. '^^  During  this  high  growth  period,  ftinding  allocations  for  higher  educafion 
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relied  primarily  on  full-time  student  enrollment  figures,  which  were  relatively  easy  to 
calculate. '^^ 

One  problem  with  focusing  performance  measurement  on  the  inputs  of 
postsecondary  education  was  that  inputs  were  given  the  "highest  importance,  followed  by 
efficiency  and  then  effectiveness."'^'* 
Process  indicators 

Process  indicators  focused  on  the  actual  delivery  of  educational  services, '^^ 
"represent[ing]  behaviors  associated  with  desired  outcomes  of  college  and  estimat[ing] 
the  extent  to  which  students . . .  engaged  in  these  activities." '^^  Process  indicators 
included  such  measures  as  the  quality  of  general  education  course  delivery,  the  length  of 
waiting  lists  for  required  courses,  faculty  evaluations,  and  the  "amounts  of  money,  time, 
and  human  resources  that  [were]  allocated  to  organizational  units  and  endeavors."'" 

Burke  and  Serban  reported  that  the  reason  more  than  40  percent  of  state-selected 
Pis  were  process  indicators  was  because  many  indicators  that  focused  on  the  outputs  of 
higher  education  were  difficuh  to  quantify. '^^  In  addition,  Kuh,  Pace,  and  Vesper 
reported  that  "process  indicators  . . .  [were]  less  difficult  and  expensive  to  develop  and 
administer  compared  with  most  outcomes  assessment  instruments."'^' 
Outcomes  indicators  ,  .  ^ 

Although  input  (or  resource)  indicators  received  much  attention  throughout  the 
decade  of  the  1 970s,  "the  1 980s  saw  a  groundswell  of  interest  in  the  other  side  of  the 
ledger:  outcomes."'^''  Serban  noted  that  the  Pis  of  the  decade  of  the  1990s  "reflect[ed]  a 
shift  of  performance  funding  from  the  resources  needed  by  higher  education  to  the  results 
produced."'^' 
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Burke  and  Serban  reported  that  other  outcomes  indicators  included  such  things  as 
student  transfers,  time-to-degree,  teaching  loads,  licensure  test  scores,  and  job  placement 
indicators.  A  survey  of  state  and  campus  leaders  showed  that  respondents  considered 
retention  and  graduation  rates,  job  placement,  licensure  exam  scores,  and  employer 
surveys  the  most  appropriate  Pis.' 

In  many  ways,  Tennessee  led  the  movement  for  reporting  outcome  information 
about  postsecondary  institutions.'^^  "Following  the  advent  of  performance  funding  in 
Tennessee,  which  specified  that  institutions  would  gather  detailed  [outcome]  information 
. . .  three-quarters  of  the  states  adopted  polices  that  caused  public  colleges  and 
universities  to  collect  and  report  some  kind  of  outcome  information."'^''  This  system, 
which  held  colleges  and  universities  accountable,  required  that  "institutions  identify  and 
measure  the  outcomes  or  products  or  effects  of  higher  education  and  then  compare  the 
results  with  the  costs."'^^  While  varying  forms  of  performance  assessment  techniques 
existed  around  the  country,  only  outcomes  assessments  had  been  widely  adopted  by  state 
and  local  educators.'^* 

Benchmarking 

In  the  decade  of  the  1990s,  benchmarking  became  a  popular  way  for  institutions 
to  evaluate  their  own  performance  relative  to  that  of  state  and  national  peer  institutions 
and  to  their  own  past  performance.'"  However,  because  it  was  preferable  for  institutions 
to  choose  Pis  closely  aligned  with  their  individual  missions,  a  question  arose  as  to  the 
"validity  of  using  Pis  to  compare  institutions."'^^ 

Ball  and  Wilkinson  worried  that  "indicators  for  comparative  purposes  might 
destroy  diversity. "'^^  These  researchers  reported  that  any  attempt  to  use  Pis  uniformly 
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across  institutions  was  likely  to  impede  the  original  purpose  of  this  type  of 
measurement — to  achieve  institutional  excellence. '^^  In  1989,  Page  summarized  the 
problem  this  way:  "[PJerformance  indicators  [were]  invaluable  . . .  tools  of  management 
information  within  institutions;  they  [were]  misleading  and  unhelpful  tools  of  political 
analysis  between  institutions — unfortunately  the  latter  [were]  more  exciting."' Mingle 
considered  benchmarks  inappropriate  for  assessing  the  performance  of  higher  education 
institutions,  since  "benchmarks  such  as  high  graduation  or  pass  rates  reveal  [ed]  little 
about  how  such  feats  were  achieved."' 

To  remedy  this  problem.  Ball  and  Wilkinson  developed  what  they  termed  "macro- 
indicators."'^^  These  indicators  "relate[d]  only  to  the  overall  system  and  not  to  individual 
institutions  within  the  system." '^"^  Because  a  degree  of  public  accountability  was 
necessary  when  talking  about  publicly  funded  entities,  macro-indicators  included  such 
things  as  "scale  and  effectiveness,  level  of  resourcing,  efficiency  and  source  of  funds."'" 

A  less  traditional  use  of  benchmarking— as  a  means  of  establishing  self-imposed 
"markers"  of  good  performance — became  more  popular  in  the  early  1990s.'^^  For 
example,  in  1992,  Tennessee  moved  from  five  to  ten  performance  funding  indicators  and 
added  several  criteria  that  rewarded  institutions  for  improvement  against  their  own 
benchmarks  on  locally  developed  goals. '    Benchmarking  was  seen  as  a  way  to 
"contextualize"  what  would  otherwise  have  been  absolute  measures  of  institutional 
performance.'^^ 

Number  of  Indicators  Used 

The  number  of  Pis  used  for  postsecondary  institutional  assessment  varied  from 
state  to  state.  According  to  Burke  and  Serban,  the  number  of  indicators  used  within 
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states  ranged  from  five  to  thirty-seven,  although  the  numbers  averaged  around  seven  to 
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Sixteen. 

Perceptions  about  the  similarity  of  Pis  used  by  states  varied  throughout  the 
literature. Some  research,  such  as  the  Rockefeller  Institute's  November  1996  state 
survey,  reported  great  variation  in  states'  choices  of  Pis. The  NCRP  also  noted  a  great 
variation  from  state  to  state  in  the  number  of  indicators  used.'^^  However,  two  indicators 
were  used  in  almost  all  state  performance  budgeting  and  funding  programs. '^^ 

The  Most  Common  Measures  of  Institutional  Performance: 
Graduation  and  Transfer  Rates 

"The  most  commonly  used  measures  for  performance  reporting  [were]  primarily 
quantitative  indicators  of 'outcome'  or  'output.'"'^"  According  to  Burke  and  Serban, 
"[0]nly  retention  and  graduation  rates  appear[ed]  as  indicators  in  nearly  all  states' 
programs."'*^  The  number  of  Pis  used  by  states  ranged  from  thirty-seven  in  South 
Carolina,  to  five  in  Minnesota,  but,  according  to  Rupert,  the  two  most  common  indicators 
of  performance  used  by  states  in  1997  were  graduation  and  transfer  rates. 

Even  though  graduation  and  transfer  rates  reflected  only  one  component  of  most 
assessment  models — individual  student  success — ^they  were  the  most  heavily  emphasized 
assessment  indicators,  used  in  thirty-two  and  twenty-five  states,  respectively.  The  State 
Higher  Education  Executive  Officers  (SHEEO)  reported  that  graduation  rates  were  most 
commonly  used  for  consumer  information,  followed  by  degrees  awarded,  admission 
standards,  and  transfer  rates. These  measures,  as  with  all  of  the  most  commonly  used 
performance  measures,  were  "primarily  quantitative  indicators  of  'outcome'  or 
'output'."'^^ 
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The  Rockefeller  survey  found  that  state  policymakers  and  postsecondary 
institution  leaders  shared  a  view  of  appropriate  indicators  of  institutional  performance.'*' 
Both  groups  agreed  that  retention  and  graduation  rates,  among  other  outcome  measures, 
were  appropriate  measurements  of  institutional  performance. 

Accuracy  of  Performance  Measures 

Some  of  the  literature  criticized  the  use  of  graduation  and  transfer  rates  as 
indicators  of  institutional  effectiveness.'^'  A  study  by  the  National  Center  for  Research 
in  Vocational  Education  (NCRVE)  presented  evidence  that  raised  doubt  about  the 
appropriateness  of  graduation  rates  as  an  effective  indicator.''^  The  NCRVE  study 
showed  that  sub-baccalaureate  higher  education,  including  course  work  that  did  not  result 
in  a  degree,  had  very  significant  economic  benefits  in  certain  labor  market  fields.''^  This 
finding  questioned  the  use  of  graduation  rates  as  an  appropriate  measure  of  institutional 
quality,  since  non-completers  (students  who  did  not  complete  a  degree  or  certificate 
program)  benefited  from  brief  enrollment  in  higher  education.'^'*  Put  another  way,  the 
economic  benefit  of  even  some  participation  in  postsecondary  education  meant  that 
graduation  rates  were  an  inappropriate  indicator  of  institutional  effectiveness."^ 

Another  criticism  of  the  use  of  graduation  and  transfer  rate  indicators  came  from 
the  American  Association  of  Community  Colleges  (AACC).''^  The  AACC  cautioned 
that  the  reporting  of  graduation  and  transfer  rates  in  isolation  disregarded  important 
aspects  of  student  life,  including  student  intentions  and  course-taking  behavior.''^  The 
AACC  suggested  focusing  on  data  reflecting  student  obtainment  of  employment  with  the 
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benefit  of  some  college  course  completion,  but  not  necessarily  the  completion  of  a  degree 
or  certificate  program. 

Astin's  research  showed  that  legislation  such  as  the  Student  Right  to  Know 
Act,'^^  which  "focus[ed]  entirely  on  student  'outcome'  measures  such  as  graduation  and  . 
. .  ignor[ed]  the  student's  original  'input,'  . . .  create [d]  a  strong  disincentive  for 
institutions  to  recruit  under-prepared  students,  not  to  mention  poor  students  and  those 
from  underrepresented  minority  groups."^*'" 

Varying  Definitions 

Even  the  most  pervasive  outcomes  measures,  such  as  retention  and  program 
completion  rates,  were  calculated  and  reported  differently  across  institutions  and 
systems.     This  lack  of  standardization  in  outcomes  data,  which  made  comparisons 
across  institutions  and  systems  virtually  impossible  to  ascertain,  reduced  the  utility  of 
outcomes  data  for  policymaking  and  evaluation.^''^ 

For  example,  the  Center  for  the  Study  of  Community  Colleges  (CSCC)  adopted 
one  of  many  possible  definitions  of  "transfer  student."^^^  It  defined  transfer  rate  as  "all 
students  entering  the  two-year  college  in  a  given  year  who  ha[d]  no  prior  college 
experience  and  who  complete[d]  at  least  twelve  college  credit  units  within  four  years, 
divided  into  the  number  of  that  group  who  t[ook]  one  or  more  classes  at  a  university 
within  four  years."^"^^  The  National  Center  for  Education  Statistics  (NCES),  on  the  other 
hand,  defined  transfer  student  in  their  Beginning  Postsecondary  Students  study 
(BPS:90/94)  as  a  student  who  "changed  institutions  permanently  without  returning  to  the 
original  institution."-^"^ 
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Astin  agreed  that  "the  relative  'effectiveness'  of  different  institutions  [could]  be 
substantially  affected  by  the  particular  definition  of  retention  used."^°^  Bray  and  Kanter 
pointed  out  that  the  adoption  of  proper  measures,  as  well  as  the  definition  of  the  measures 
adopted,  were  central  themes  of  the  performance  measure  debate.'^^^ 

Performance  Indicator  Developers 

The  selection  of  Pis  and  the  adoption  of  performance  funding  and  budgeting  were 
subject  to  the  interpretations  of  those  who  developed  and  adopted  them.^°^  Serban 
reported  three  ways  in  which  performance  funding  could  be  adopted:  (1)  state-mandated 
performance  funding  and  legislatively  prescribed  goals  and  indicators;  (2)  state-mandated 
performance  funding,  and  higher  education  coordinating  board-prescribed  indicators;  and 
(3)  performance  fiinding  initiated  by  higher  education  coordinating  bodies  to  prevent  a 
governmental  mandate. 

In  Colorado,^"^  Minnesota,^'^  Ohio,^' '  and  South  Carolina,^'^  for  example, 
"legislation  mandated  the  program  and  prescribed  the  indicators"  to  be  used.^'^  Ruppert 
reported  that  most  accountability  programs,  as  conceived  in  state  policy,  were  "developed 
by  state  policymakers  for  state  policymakers."  (Emphasis  added.)^"* 

In  other  states,  however,  legislation  mandated  the  initiation  of  performance 
funding  programs,  but  left  the  higher  education  coordinating  agencies  and  campus  leaders 
to  propose  the  specific  Pis  to  be  used.^'^  NCRP  reported  that  the  more  effective 
programs  allowed  institutions  to  participate  in  the  selection  of  the  indicators.^ 

Some  states  used  a  combination  of  statewide  and  institution-specific  indicators.^" 
For  example,  in  1997,  six  states  set  statewide  indicators  for  use  by  all  institutions  (both 
two-  and  four-year)  of  which  three  permitted  campus-specific  indicators.^ 
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Still  other  methods  for  deriving  indicators  were  used  in  Tennessee,^'^  Arkansas/^" 
and  Kentucky,    where  coordinating  agencies  worked  with  institutional  representatives 
to  implement  performance  fiinding  programs  without  legislative  interference.^^^ 
According  to  Burke  and  Serban,  the  means  by  which  performance  funding  was  initiated 
within  a  state  "influence[d]  the  critical  choices  of  performance  indicators  and  their  budget 
weights,  success  standards,  and  fiinding  levels."'^^'^ 

Finally,  California's  "Partnership  for  Excellence"^^''  was  the  result  of  the  "first 
higher  education  governing  board  in  the  nation  to  propose  a  performance  funding 
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program."     No  other  Board  of  Governors  to  date  had  proposed  a  performance  funding 
plan.22^ 

Community  College-Specific  Performance  Indicators 

"Demonstrating  institutional  effectiveness  present[d]  a  special  set  of  problems  for 
community  colleges.""^^^  According  to  Walleri  and  Seybert,  Pis  were  built  on  the  four- 
year  college  and  university  paradigm,  and  "community  college  students  oflen  differ[ed] 
dramatically  from  traditional  college  students.  The  former  [were]  much  more  diverse  in 
age,  background,  employment  status,  preparation,  and  educational  objectives  than  their 
four-year-college  counterparts."^'^^  "[M]easures  of  institutional  effectiveness  common  to 
four-year  colleges  and  universities  . . .  [were]  in  most  cases  not  applicable  to  community 
colleges."^' 

Walleri  and  Seybert  also  reported  that  community  college  assessment  strategies 
included  efforts  to  "demonstrate  institutional  effectiveness,  including  a  strong  outcomes 
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assessment  component."     According  to  these  researchers,  other  important  components 
of  an  assessment  strategy  included:  "systematic  program  reviews,  follow-up  surveys  of 
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former  vocational  students  and  their  employers,  follow-up  surveys  and  senior  institution 
transcript  analyses  of  former  transfer  students,  periodic  evaluation  surveys  of  college 
offices  and  services,  and  analyses  of  course  retention  and  attrition  rates  and  grading 
patterns.""' 

State  Use  of  Two-Year  College  Indicators  of  Performance 

With  the  exception  of  California,"^  Florida,"^  Missouri,""  Colorado,"^  and 
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Ohio,    by  1998  very  few  states  had  developed  Pis  for  use  specifically  in  the  community 
college  sector.^^^  Although  these  states  had  developed  community  college-specific 
indicators,  Ohio's^^^  performance  funding  applied  only  to  two-year  colleges — the  state's 
four-year  colleges  were  not  performance  funded  at  that  time.^^^ 

Although  the  five  states  noted  above  had  developed  community  college-specific 
indicators  by  1998,  the  use  of  these  indicators  varied.  For  example.  Burke  and  Serban 
reported  that,  while  Florida^''^  had  completely  separate  measures  for  its  community  and 
four-year  college  sectors  at  that  time,  several  states  provided  only  one  alternate  item  for 
use  in  conjunction  with  a  common  set  for  all  higher  education  institutions  in  the  state. 
Missouri^"*^  was  one  of  the  only  states  to  have  developed  a  specific  set  of  indicators  for 
the  community  college  sector,  which  was  used  in  conjunction  with  a  set  of  indicators 
common  to  all  higher  education  institutions.'^'*^ 

The  focus  of  state  Pis  also  varied.  For  instance,  Ohio's  adoption  of  community 
college-specific  Pis  focused  on  service  expectations  of  the  state,  such  as  non-credit 
continuing  education  opportunities,  partnerships  with  business  and  industry,  community 
involvement,  linkages  with  high  schools,  and  student  access— including  affordability  and 
convenient  scheduling.^'*'* 
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California  was  one  state  that  developed  community  college  PIs?^'  A  state  budget 
signed  by  Governor  Pete  Wilson  in  August  of  1998  earmarked  $100  million  for  a  system- 
wide  community  college  accountability  program — Partnership  for  Excellence.'^''^  This 
program  included  about  $300  million  more  for  California's  106  community  colleges,  and 
focused  on  performance  outcomes  including  transfer,  degrees  and  certificates,  successful 
course  completion,  workforce  development,  and  basic  skills  improvement.^'*'  Two  of  the 
five  California  community  college  indicators — transfer  and  graduation  rates — were 
among  the  most  commonly  used  in  all  states?''^ 

The  California  plan  also  recognized  and  validated  remediation  as  a  key 
component  of  the  community  college  mission.^''^  In  so  doing,  the  plan  enabled        .  '  ' 
California's  community  colleges  to  adopt  measures  appropriate  for  this  area  of 
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mstruction.  » 

Recognizing  that  community  college  indicators  of  success  might  differ  from  those 
appropriate  to  four-year  colleges  and  universities,  Colorado  adopted  separate 
performance  indicators  for  its  two-year  sector.^^'  Colorado  borrowed  extensively  from 
the  American  Association  of  Community  College's  (AACC)  1994  Core  Indictors  of 
Effectiveness?^^ 

Mission-Specific  Community  College  Indicators 

Although  many  states  did  not  have  separate  indicators  for  community  colleges, 
many  adopted  priorities  that  reflected  the  mission  of  community  colleges.^"  In  1998, 
"the  Louisiana  legislature  approved  a  constitutional  amendment  that  create[d]  a  fifty- 
college  system  aimed  at  building  the  state's  workforce."^^''  This  legislation  effectively 
made  workforce  development  its  top  priority,  and  included  separate  performance-based 
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financing  formulas  for  two-year  colleges?"  The  community  college-specific  formulas 
emphasized  measures  such  as  job  placement  and  salary  rates  of  graduates,  and  the 
responsiveness  of  colleges  to  industry  priorities.^^^ 

In  1 991 ,  Florida  began  requiring  the  state's  twenty-eight  community  colleges  to 
report  annual  graduation  rates,  student  progression,  minority  retention,  placement  rates, 
and  minority  enrollments.'^"  By  1994,  Florida  had  revamped  its  program  of  institutional 
evaluation  by  using  the  state  community  college  system's  mission  statement  as  a 
framework.^^^ 


Selection  of  Community  College-Specific  Indicators 

The  AACC's  1994  Core  Indicators  of  Effectiveness  provided  a  widely  used  model 
for  the  development  of  community  college  PIs?^^  The  AACC  deemed  the  following 
indicators  as  appropriate  for  the  community  college  sector: 

1 .  Student  goal  attainment 

2.  Persistence  (fall-to-fall) 

3.  Degree  completion  rates 

4.  Workforce  placement  rates 

5.  Employer  assessment  of  students 

6.  Number  and  rate  of  transfers 

7.  Performance  after  transfer 

8.  Success  in  subsequent  related  coursework 

9.  Demonstration  of  critical  literacy  skills 

10.  Demonstration  of  citizenship  skills 

1 1 .  Client  assessment  of  programs  and  services 

12.  Responsiveness  to  community  needs 

1 3 .  Participation  rate  in  service  area 

Although  there  was  a  void  in  the  literature  pertaining  directly  to  community 
college  performance  funding  and  budgeting,  prominent  researchers  in  this  area  agreed  on 
the  selection  of  appropriate  performance  indicators  for  this  sector.^^**  According  to 
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Borden  and  Banta,  for  example,  "in  the  area  of  two-year  college  transfer  effectiveness  . . . 
appropriate  measures  of  performance  from  the  institutional  perspective  [were]  transfer 
rates  and  the  performance  of  former  students  at  senior  institutions."^^' 

Serban  stressed  a  need  to  "reconsider  the  success  criteria  used  to  judge 
institutions'  progress  on  performance  funding  indicators,  and  a  desire  for  the  use  of 
combinations  of  institutional  improvement  over  time,  comparisons  to  peer  institutions, 
and  targeted  external  standards."'^^^ 

Developing  Performance  Indicators 

The  literature  on  PBF  and  PBB  underscored  the  controversy  surrounding  the 
misuse  of  performance  measures.'^^^  "As  state  and  federal  interest  in  the  use  of  indicators 
[grew],  it  [was]  increasingly  apparent  that  an  indicator  must  be  more  than  simply  a 
readily  calculable  statistic."^^ 

Albright  recommended  the  following  key  principles  for  developing  and 

implementing  PBF«S:B:^^^ 

Embrace  a  clear  vision  and  common  goals;  achieve  real  reform  through  an 
improvement  approach;  connect  performance  funding  with  strategic  planning; 
tailor  the  program  to  meet  each  state's  culture  and  each  institution's  educational 
needs;  promote  collaboration;  promote  open  communications  and  an  inclusive 
process;  expect  trial-and-error  periods  and  mistakes;  begin  with  a  pilot  effort,  and 
develop  the  implementation  plan;  before  final  decision  making  and  marketing; 
limit  the  size  of  the  budget  allocated  to  performance  funding;  use  measures  that 
reflect  quality  and  customer  focus  rather  than  numeric  counts;  use  existing  data 
whenever  possible;  to  the  extent  possible,  focus  on  measures  within  the  control  of 
the  institution.^^^ 

Most  of  the  suggestions  regarding  performance  indicator  development  and 
adoption  focused  on  two  broad  areas:  flexibility  of  indicators  and  size  of  budget 
allocations.'^^^ 
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Flexibility:  Allow  for  Mission  Diversity 

According  to  the  NCRP,  one  of  the  challenges  of  designing  a  performance 
fiinding  or  performance  budgeting  system  was  in  "finding  measures  that  adequately 
reflect[ed]  differences  in  institutional  mission."^^^  Burke  and  Serban  agreed  that 
indicators  common  to  all  state  higher  education  institutions  needed  to  be  used  in 
conjunction  with  separate  categories  for  two-  and  four-year  institutions  to  "emphasize 
their  different  missions  without  neglecting  common  goals."^^^  Ball  and  Wilkinson  also 
suggested  a  focus  on  mission.^^° 

SHEEO  reported,  "[SJtandardizing  state  goals  for  diverse  institutions  d[id]  not 
work;  balancing  system-wide  goals  with  unique  or  customized  goals  or  measures  [was]  a 
more  effective  strategy.""'  SHEEO  stressed  that  more  emphasis  should  be  placed  on 
part-time  students  and  workforce  training^'^ — common  components  of  the  community 
college  mission. 

According  to  Ball  and  Wilkinson,  "Higher  education  institutions  [were]  not 
football  teams,  they  ha[d]  differing  institutional  missions  and  operate[d]  in  different 
environments.""^  Mission  diversity  was  "clearly  beneficial,"  since  it  allowed  for  "a 
range  of  institutions  that  [would]  give  a  genuine  choice  to  prospective  students  in 
particular  and  society  in  general."'^^'* 

State  Use  of  Flexible  Indicators 

In  California,"^  among  other  states,  the  PBF  package  gave  college  administrators 
"flexibility  in  choosing  which  outcomes  fit  their  institutions.""^  California's  plan  gave 
individual  colleges  the  flexibility  to  choose  goals  that  matched  their  missions,  as  well  as 
"the  freedom  to  spend  the  money  to  achieve  the  goals  they  set  for  themselves."^" 
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South  Carolina's  state  commission  also  allowed  for  mission  diversity  when  it 
created  four  versions  of  its  performance-based  financing  system  with  separate  categories 
for  research  universities,  teaching  universities,  technical  colleges,  and  branches  of  the 
University  of  South  Carolina.      The  state  commission  did  not,  however,  develop  a 
separate  category  for  community  colleges. 

Levels  of  Funding  Allocations 

The  NCRP  recommended  tying  only  a  small  portion  of  the  state  budgets  to 
performance  funding  or  budgeting.  It  reported  that  the  most  effective  programs  generally 
tied  less  than  5  percent  of  the  college's  budget  to  performance  budgeting  and  funding, 
thereby  "reward[ing]  successes  rather  than  penalizing  failures."^^^  Ewell  concurred  that 
"[IJinking  high  stakes  and  measured  performance  c[ould]  have  negative  consequences  . . . 
[because]  if  the  stakes  [were]  high,  campuses  [could]  start  'managing  to  the  numbers.'"^**' 
Albright  reported  that  2  to  5  percent  of  appropriations  could  "effect  meaningful  change 
without  risking  major  budget  instability."^^' 

On  the  other  hand,  defining  the  amount  of  funds  allocated  through  performance 
funding  and  budgeting  required  a  delicate  balance.  Ewell,  for  instance,  pointed  out  that  if 
the  stakes  were  too  low,  they  risked  being  ineffective  in  creating  change.^*^ 

Overall,  there  was  no  prescribed  balance  of  allocations  that  could  be  used  across 
states  and  institutions.  Albright  pointed  out  that  what  worked  well  in  some  states  might 
not  have  worked  as  well  in  others.^^^  "To  assume  [that]  any  single  accountability  system 
w[ould]  definitely  fulfill  the  expectations  of  all  of  the  actors  . . .  [was]  to  assume  a  system 
that  [had  not]  been  invented  yet."^^"* 


Advantages  of  Performance-Based  Funding  and  Budgeting  (PBF&B) 

Proponents  of  PBF&B  pointed  out  several  characteristics  that  made  this  funding 
system  superior  to  past  funding  practices?*^  According  to  Borden  and  Banta, 
performance-based  allocations  were  effective  fiinding  methods  because  they  offered 
"explicit  points  of  reference,"  whereas  past  methods  had  not?^^  According  to  Bray  and 
Kanter,  PBF&B  offered  rewards  and  incentives,  without  which  postsecondary  education 
would  not  have  been  motivated  to  make  positive  changes  to  improve  performance.^^^ 

Albright  reported  that  PBF&B  provided  a  way  for  higher  education  to  regain  the 
support  of  political  leadership  by  providing  evidence  of  postsecondary  education's 
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quality.     Similarly,  the  NCRP  reported  that  performance  funding  served  as  an  incentive 
for  improved  performance;  responded  to  calls  for  accountability;  connected  plans  with 
budgets;  improved  the  image  and  credibility  of  colleges  and  universities;  improved 
postsecondary  education's  communication  with  political  leaders;  and  was  more  effective 
than  considering  only  inflation  and  enrollment  growth  in  funding  decisions.'^^^ 

Performance  measures  served  both  budgetary  and  institutional  planning  purposes. 
According  to  Ball  and  Wilkinson,  Pis  "provide[d]  a  more  coordinated  and  integrated  - 
approach  to  institutional  decision  making."'^^'*  "[P]erformance  measures  [were]  used  to: 
eliminate  programs;  reduce  the  size  of  programs;  focus  on  outcomes  rather  than  inputs; 
clarify  state  priorities;  emphasize  important  state  educational  goals;  provide  better 
accountability  information;  assess  the  effective  utilization  of  resources;  link  budgeting 
with  planning  and  programming;  promote  improvement;  reinforce  existing  quality; 
promote  assessment;  and  encourage  institutional  change."^^'  - 
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Problems  with  Pis  and  Performance-Based  Funding  and 
Performance-Based  Budgeting  (PBF&B) 

Performance  funding  was  called  "one  of  the  most  controversial  policies  of  the 
1990s,"  due  to  the  difficulty  of  selecting  of  appropriate  PIs?^^  In  most  cases,  PBF&B 
were  not  at  the  heart  of  the  controversy;  usually  the  selected  Pis  and  their  interpretations 
fueled  the  debate.  Burke  and  Serban  reported  that  the  "selection  of  performance 
indicators  and  success  standards,  and  protection  of  mission  diversity  [were]  seen  as  the 
major  difficulties  associated  with  performance  funding."^'^ 
Clarity  of  purpose 

Pis  should  have  included  a  clear  definition  of  whether  the  levels  should  go  up  or 
down  to  indicate  the  desired  level  of  performance.^^'*  For  example,  problems  occurred 
when  colleges  were  evaluated  on  conflicting  indicators  such  as  SAT  scores  and 
accessibility.  Although  high  SAT  scores  indicated  institutional  success,  accessibility 
necessitated  lower  SAT  entrance  requirements.^^^ 
Subjective  development 

According  to  Banta  and  Borden,  "Whoever  determine[d]  the  performance 
indicators  determine[d]  the  activities  and  direction  of  the  system,  institution,  or 
program."^^^  In  1997,  for  example,  the  chairman  of  the  South  Carolina  Council  of 
Public  College  and  University  Presidents  reported  that  "institutions  [were]  at  the  mercy 
of  legislative  leaders  who  [knew]  little  about  higher  education  and  [did]  not  appreciate 
how  difficult  it  [was]  to  define,  much  less  measure,  what  a  'good'  public  college 
[did]."2^^ 
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Counterproductive  use 

Pis  could  have  become  incentives  for  counterproductive  actions.     For  example, 
reporting  requirements  under  the  Student-Right-to-Know  Act  and  the  National  Collegiate 
Athletic  Association  (NCAA)  focused  on  six-year  graduation  rates  among  cohorts  of 
entering  freshmen  in  four-year  institutions.^^^  This  may  have  provided  an  incentive  for 
colleges  and  universities  to  change  their  admissions  requirements  and  programs  simply  to 
improve  their  level  of  performance  on  this  particular  indicator,  despite  what  it  might  have 
done  to  other  areas  of  non-measured  performance.^^*^ 
Oversimplification 

There  was  also  a  danger  of  oversimplification  or  taking  a  reductionist  approach — 
reducing  one's  goals  to  what  one  could  measure.  Ruppert  reported  that  indicators  should 
"reflect  goals  and  purposes"  and  not  just  be  collected  "because  the  data  [was] 
available."     Despite  the  danger  of  oversimplification,  much  of  the  literature  agreed  that 
the  PI  approach  was  appropriate,  because  the  "higher  education  process  [was]  one  of 
transforming  inputs  into  outputs  having  a  higher  value."^"^ 
Mission  inconsistency 

According  to  Schmidt,  "Many  institutions  [were]  afi-aid  that  they  [would] 
immediately  begin  losing  millions  of  dollars  to  a  system  that  [was]  biased  against  them  or 
fail[ed]  to  take  into  account  their  unique  character  and  needs."^*'^  This  could  have 
presented  a  problem  for  institutions  whose  missions  were  more  oriented  toward  part-time, 
older,  or  working  students,  who  had  been  characterized  as  higher  education's  "new 
majority."^"^  The  NCRP  agreed  that  finding  measures  to  adequately  reflect  differences  in 
institutional  mission  was  difficult.^*^^ 


51 


Measurement  difficulties 

Bowen  noted  that  "outcomes  [were]  extremely  hard  to  isolate  and  measure."^*^ 
According  to  the  NCRP,  "addressing  the  complexities  of  measuring  quality,  particularly 
student  learning,"  posed  a  particular  problem  for  postsecondary  use  of  PBF  and  PBB. 
"[M]any  academics  beiieve[d]  that  the  most  important  outcomes  of  education  [were] 
simply  not  measurable."^"^ 

There  was  little  agreement  about  which  outcomes  were  important  or  how  they 
should  be  measured.^°^  "The  time  and  effort  involved  in  designing  the  measures, 
collecting  the  relevant  data,  and  analyzing  the  results  [were]  so  great  as  to  discourage 
most  investigators  from  tackling  the  problem."^    Ruppert  pointed  out  that  many  Pis  did 
not  take  into  account  what  the  student  wanted  or  expected  from  higher  education.^" 
Despite  Ruppert's  emphasis  on  the  importance  of  knowing  what  students  thought  and 
wanted,  student  input  in  the  development  of  Pis  at  the  state  level  was  scarce.  ^'^ 
Instability 

The  yearly  budget  cycle  and  the  turnover  of  state  legislators  created  an 
environment  of  instability  for  most  public  universities  and  colleges.  The  NCRP  reported 
that  the  maintenance  of  institutional  autonomy  with  state-level  control,  and  the 
adjustment  to  annual  changes  in  legislative  priorities  presented  unique  challenges.^'^ 
Serban  echoed  these  concerns.^''* 
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The  Future  of  Performance-Based  Funding  and 
Performance-Based  Budgeting  (PBF4&B) 

Langfitt  warned  that  "higher  education  [was]  losing  public  trust,  and  [the 
institutions]  must  either . . .  respond  more  effectively  to  their  publics  or . . .  face  even 
more  government  intervention  in  their  affairs."^ 

Banta  reported  that  most  faculty  "resent[ed]"  externally  imposed  performance 
measures,  and  "believe[d]  they  [would]  eventually  go  away."^'^  A  survey  conducted  in 
1997,  however,  revealed  that  "state  policymakers  tend[ed]  to  forecast  a  favorable  future 
for  performance  funding,"  despite  college  officials'  belief  that  it  faced  an  uncertain 
future.^"' 

Although  much  of  the  literature  indicated  that  PBF&B  were  likely  to  be  adopted 
by  more  states  in  the  future,  the  uses  of  the  Pis  appeared  to  be  shifting.^ For  example, 
the  current  and  fiiture  use  of  Pis  for  budgeting  purposes  was  clearly  articulated  by  the 
findings  of  a  1996  survey  conducted  by  the  Rockefeller  Institute:  "[E]xtemal 
accountability  was  the  . . .  main  purpose  of  performance  funding  . . .  [however],  all 
groups  wish[ed]  to  see  institutional  improvement  become  the  primary  purpose."-^ 

Toward  an  Input-Based  Performance  Indicator 

Spanning  decades  of  research,  Alexander  Astin  produced  a  comprehensive  body 
of  literature  dealing  with  the  prediction  of  student  academic  success.^^^  His  most  recent 
works  focused  on  the  applicability  of  his  prior  student  success  research  to  the 
accountability  wave  of  the  decade  of  the  1990s.^^'  Asfin  reported  in  1993  what  other 
researchers  agreed  to  be  the  inadequacies  of  absolute  Pis:  "[A]  simple  retention  'rate' 


[told]  us  a  lot  more  about  who  an  institution  admit  fed]  than  about  how  effective  its 
retention  practices  [were]. "^^^  '  '  '' 

This  finding  was  analogous  to  the  problems  with  other  Pis  that  relied  solely  on 
absolute  rates — graduation,  completion,  retention,  and  passing  rates.  Since  "[i]nstitutions 
that  admit[ed]  large  numbers  of  less-well-prepared  students  tend[ed]  to  have  low 
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retention  rates,"    and  the  open-admissions  policies  of  community  colleges  attracted 
students  with  lower  levels  of  preparation,  it  followed  that  community  colleges  also  had 
lower  graduation  rates  than  their  four-year  counterparts. 

Astin's  findings  indicated  that  since  "more  than  half  of  the  variation  in  retention 
rates  [which  is  directly  related  to  graduation  rates] . . .  [could]  be  explained  by  [an 
institution's]  expected  retention  rates,  more  selective  institutions  [would]  have  'good' 
retention  rates  even  if  their  retention  programs  [were]  mediocre,  and  most  non-selective 
institutions'  rates  [would]  be  'poor'  unless  they  happen[ed]  to  have  exceptionally 
effective  retention  programs."^^''  This  problem,  as  presented  by  Astin,  was  exactly  the 
one  faced  by  less  selective  schools  when  absolute  measures  of  institutional  performance 
were  used  to  allocate  funds.  Astin  believed  it  was  "unfair"  to  compare  retention  rates  of 
different  types  of  institutions.^"  Astin  focused  instead  on  how  well  institutions 
performed  given  the  students  they  admitted.^^^ 

The  use  of  retention  rates  as  an  indicator  of  institutional  effectiveness  made  it 
necessary  to  take  into  account  differences  in  entering  student  characteristics  (inputs) 
when  comparing  the  retention  rates  (outputs)  of  colleges  and  universities.^^^  Using  a 
statistical  regression  model,  Astin  demonstrated  how  a  college  with  a  relatively  low 
retention  rate  that  admitted  students  who  were  not  well  prepared  for  college-level  studies 
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could  be  viewed  as  performing  better  than  a  college  with  a  significantly  higher  retention 
rate  that  had  more  selective  admissions  standards. 

Similarly,  Mortenson  utilized  regression  analysis  to  predict,  among  other  things, 
the  graduation  rates  and  student  persistence  rates  of  four-year  college  students  and 
institutions.      His  work,  while  contributing  to  the  large  body  of  literature  focused  on 
four-year  college  student  and  institutional  success,  did  not  focus  on  the  factors  that 
contributed  to  the  success  of  community  college  students.''^^ 

Although  Mortenson's  use  of  regression  analysis  for  the  prediction  of  student 
behavior  followed  a  long  history  of  this  type  of  research,  his  use  of  these  regression 
models  was  slightly  different.      For  instance,  through  his  use  of  data  from  U.S.  News 
and  World  Report,"^  IPEDS,"^  ACT,""*  and  others,"^  Mortenson  developed  a  basic 
model  for  the  prediction  of  institutional  graduation  rates.^''^  .  * 

Mortenson's  regression  analyses  indicated  that  the  "most  important  factors 
explaining  the  differences  in  [graduation  rates]  between  institutions  [were]  the  pre- 
college  academic  backgrounds  students  br[ought]  with  them  when  they  enrolled]  in 
college.""^  Specifically,  high  school  grades  and  college  admissions  test  scores 
demonstrated  powerful  effects  over  the  "rate  at  which  freshmen  graduate[d]  from  their 
college  of  first  enrollment  as  well  as  their  time-to-degree.""^  Even  when  these  pre- 
college  characteristics  were  held  constant,  there  was  a  "substantial  residual  in 
[institutional  graduation  rates]  that  remain[ed]  to  be  explained.""' 

Through  the  comparison  of  predicted  graduation  rates  (calculated  from  the 
application  of  institutional  data  to  the  basic  regression  model)  to  actual  graduation  rates, 
Mortenson  concluded  that  "factors  beyond  the  pre-college  academic  records  of  students 
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influence[d]  chances  for  graduating  from  college. This  method  was  called  the 
residual  or  input  approach  to  institutional  performance  assessment.  By  considering  those 
student  factors  not  under  the  control  of  the  institution — part-time  attendance  patterns, 
race,  age,  number  of  dependents,  academic  goals,  etc. — ^the  difference  between  predicted 
and  actual  graduation  rates  could  be  "attribute[d] ...  to  internal  institutional  factors  that 
either  foster[ed]  or  imped[ed]  the  rate  at  which  admitted  freshmen  complete[d]  their 
bachelor's  studies."^'*'  A  comparison  of  predicted  and  actual  retention  rates  revealed  that 
there  were  "environmental  factors  within  institutions  that  explain  [ed]  supportive  or  non- 
supportive  academic  and  social  environments  for  student  persistence  and  success."''''^ 
■  .      This  method  offered  an  "appropriate  reference  norm"  for  the  developers  of 
institutional  PIs.^"*^  Specifically,  this  method  was  a  cost-effective  way  to  measure  the 
extent  to  which  "[s]ome  institutions  d[id]  a  better  job  with  the  academic  material  they 
enroll[ed]."^^ 

According  to  Mortenson,  graduation  and  retention  rates  based  on  raw  data  were 
inappropriate  indicators  of  performance.  Although  this  type  of  raw  data  was  often  used 
in  the  ranking  of  colleges  and  universities,  "[n]ot  until  academic  and  other  external 
environmental  factors  [were]  controlled  for  [could]  reasonable  judgments  about  the 
institutional  graduation  rates  be  formed."^''^  Mortenson  warned  that  differences  between 
institutions  were  explained  largely  by  pre-college  student  variables,  and  that  any  use  of 
this  comparative  data  without  controls  for  pre-college  student  variables  was  highly 
misleading.^"*^ 

In  1990,  Johnes  and  Taylor  developed  a  method  for  the  comparison  of  Pis  across 
mstitutions,^"*^  based  on  the  same  problems  with  performance  indicator  comparability 
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cited  by  Mortenson,    Adelman,    and  Astin.     This  method  employed  regression 
analysis  to  address  some  of  the  difficulties  faced  when  comparing  institutions  of  varying 
natures. 

Of  specific  concern  to  Johnes  and  Taylor  were  the  differences  in  non-completion 
rates  across  institutions  and  the  corresponding  assumptions  based  on  these  outcomes.^^' 
To  address  the  differences  in  institutions,  several  student-related  factors  (such  as  entry 
qualifications)  and  several  university-related  factors  (such  as  percentage  of  students 
taking  business,  social  science  or  language  programs,  students  living  in  on-campus 
housing,  and  staff/student  ratio)  were  included  in  the  analysis.  What  resulted  was  a  basic 
model  of  non-completion  rates  for  the  institutions  in  the  sample,  a  model  which  could  be 
used  to  predict  the  expected  level  of  drop  out  for  each  institution  based  on  entry-level 
qualifications,  subject  mix,  and  the  percentage  of  students  living  in  halls  of  residence/on- 
campus  housing.^"  Finally,  Johnes  and  Taylor  studied  the  differences  between  actual 
drop  out  rates  and  predicted  drop  out  rates  (residuals)  as  a  measure  of  institutional 
performance. 

Predictors  of  Student  Success  and  Institutional  Performance 

A  large  body  of  research  existed  regarding  predictors  of  four-year  college  student 
success.  For  example,  Astin' s  years  of  research  on  four-year  college  student  behavior 
uncovered  variables  shown  to  predict  student  success.      Even  as  early  as  1971,  Astin 
reported  that  high  school  grades,  SAT  scores,  and  student  gender  were  strong  predictors 
of  college  success.^^"*  His  more  recent  works  showed  similar  resuhs:  "[H]igh-school 
grades  and  SAT  scores  carr[ied]  the  most  weight  in  predicting  who  [would]  complete 
college."^^^ 
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Astin's  findings  included  the  following  list  of  predictors  for  freshman  college 
grade  point  average — a  measure  of  student  success:  (1)  membership  in  a  scholastic 
honor  society;  (2)  former  enrollment  in  a  private  high  school;  (3)  high  self-confidence  in 
academic  ability;  and  (4)  high  self-confidence  in  drive  to  achieve.^^^  Another  predictor 
discovered  by  Astin  was  the  student's  academic  major;  business,  education,  and  physical 
education  majors  had  lower  freshman  grade  point  averages  (GPAs)  than  students  in  other 
fields.'''^  Astin  also  found  that  student  race,^^*  religion  (except  for  Jewish  men,  who  had 
higher  GPAs  overall),  and  parental  educational  and  economic  status  had  only  "very  small 
effects  on  ...academic  achievement  during  the  freshman  year  in  college."^^' 

Tinto's  theoretical  model  of  student  persistence/withdrawal  behavior,  which 
contained  variables  of  social  and  academic  integration,  was  the  focus  of  a  large  body  of 
research  from  the  past  twenty-three  years.^^'  Much  of  the  subsequent  research  confirmed 
Tinto's  findings  that  personal  relationships  formed  between  students  and  faculty 
members  were  strong  predictors  of  student  persistence.^^^ 

Adelman  found  that  "one's  life  course  [was]  essentially  determined  at  age 
eighteen,  the  bend  in  the  metaphoric  river  at  which  the  college  admissions  decision 
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occurs."     He  reported  that,  while  pre-college  factors  were  the  strongest  predictors  of 
degree  completion,  it  was  "neither  a  test  score  nor  a  high  school  grade  point  average  or 
class  rank"  that  made  the  difference.^^  Instead,  Adelman  concluded  that  one's  academic 
curriculum  in  high  school — in  terms  of  both  intensity  and  quality  level — was  the 
strongest  predictor  of  postsecondary  education  success. 
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Two- Year  College  Predictors 

Within  the  small  amount  of  research  of  this  kind  focused  on  two-year  colleges 
and  their  students,  Pascarella,  Smart,  and  Ethington  found  that  "person-environment  fit 
[was]  an  important  determinant  of  persistence  in  [two-year  college]  postsecondary 
education."     Specifically,  students  at  two-year  colleges  were  more  likely  to  "persist  in 
the  pursuit  of  the  bachelor's  degree  if  they  became  successfully  integrated  into  the 
academic  and  social  systems  of  the  last  institution  attended."^^^  These  researchers  found 
that  pre-college  factors  may  not  have  been  as  influential  in  student  persistence  for  two- 
year  college  students  as  they  were  for  other  cohorts.^^^ 

Conclusion 

Performance  funding  and  performance  budgeting  gained  popularity  during  the 
decade  of  the  1990s  as  a  credible  and  effective  means  of  funding  public  higher 
education.^^^  However,  the  growth  in  this  type  of  funding  generated  much  debate 
concerning  both  the  types  of  Pis  utilized  to  reflect  higher  education  institutions'  quality 
and  the  amount  of  state  budgets  to  be  allocated  in  this  manner.^^^  Most  states  that 
utilized  performance  funding  or  performance  budgeting  did  not  include  a  separate  set  of 
indicators  for  their  community  coUeges.^^*^  This  lack  of  differentiation  between  the 
indicators  for  four-year  and  two-year  colleges  created  the  potential  for  inequitable 
distribution  of  funds  to  higher  education."'^'  A  model  that  could  accurately  predict  the 
success  of  entering  college  students  (specifically,  graduation  and/or  transfer  rates  from  a 
higher  education  institution)  could  be  compared  to  actual  success  rates  at  individual 
institutions  as  a  measure  of  institutional  performance.^^^  Although  this  type  of  model 


was  developed  successfully  for  other  purposes  in  the  literature,  it  could  be  extended  to 
the  performance  funding  arena.         <  f  - 
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CHAPTER  3 
METHODOLOGY 

Introduction 

As  evidenced  throughout  the  literature,  performance  funding  and  performance 
budgeting  gained  popularity  during  the  decade  of  the  1990s  as  a  credible  and  effective 
means  of  funding  public  higher  education. '  However,  the  growth  in  this  type  of  funding 
generated  much  debate  concerning  both  the  types  of  performance  indicators  (Pis)  used  to 
reflect  higher  education  institutions'  quality  and  the  amount  of  state  budgets  to  be 
allocated  in  this  manner.^  Of  the  states  that  utilized  performance  funding  or  performance 
budgeting,  most  did  not  include  a  separate  set  of  indicators  for  their  community  colleges.^ 
This  lack  of  differentiation  between  the  indicators  for  four-year  and  two-year  colleges 
created  the  potential  for  inequitable  distribution  of  funds  within  higher  education.''  A 
model  that  could  accurately  predict  the  success  of  entering  college  students  (specifically, 
graduation  and/or  transfer  rates  from  a  higher  education  institution)  could  be  compared  to 
actual  success  rates  at  individual  institutions  as  a  measure  of  institutional  performance.^ 
This  type  of  model,  which  was  successf\illy  developed  for  other  purposes  in  the  literature, 
could  be  extended  to  the  performance-funding  arena. 

In  order  to  address  some  of  these  issues,  this  research  examined  the  following  two 
areas  of  inquiry: 
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The  first  question  was  determining  how  performance  assessment  and  performance 
measures  were  used  to  allocate  state  funds  to  postsecondary  education  institutions.  Was 
performance  assessment  used  for  all  types  of  postsecondary  institutions — both  two-year 
and  four-year  colleges  and  universities?  Which  states  had  used,  were  using,  and/or 
planned  to  use  this  type  of  fiinding  allocation?  Of  the  states  currently  using  performance 
assessment  fiinding,  what  portions  of  their  individual  budgets  did  they  allocate  in  this 
manner?  Did  states  use  multiple  measures  of  institutional  performance  to  allocate  funds, 
and,  if  so,  how  many  did  they  use?  Were  institutional  performance  measures  the  same 
for  all  postsecondary  institutions?  Specifically,  was  performance  measured  differently 
for  four-year  and  two-year  colleges  and  universities?  What  measures  of  postsecondary 
institutional  performance  did  states  most  frequently  use  to  allocate  funds?  ^  • 

This  research  question  was  answered  through  a  review  of  the  literature  and  is 
addressed  in  Chapter  2.  .  '  ■ 

The  second  question  was  whether  the  institutional  performance  measures  used  by 
states  and  institutions  accurately  measured  the  performance  of  community  and  two-year 
colleges.  Could  the  outcome  measures  most  commonly  used  by  states  and  institutions 
accurately  depict  community  and  two-year  college  performance?  How  feasible  was  the 
development  of  performance  measures  to  account  for  differences  in  levels  and  types  of 
institutions?  Specifically,  was  there  a  model  that  could  serve  as  an  accurate  performance 
indicator  for  both  two-year  and  four-year  postsecondary  institutions? 

This  research  question  was  answered  through  a  review  of  the  literature  and 
through  the  use  of  logistic  regression  analysis  to  develop  a  model  of  community  college 
performance.  See  Chapters  2  and  4,  respectively. 
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The  purpose  of  this  study  was  to  examine  the  use  of  performance  assessment  and 
performance-based  funding  and  budgeting  (PBF&B)  in  United  States  community 
colleges.  This  study  was  designed  (1)  to  determine  whether  current  models  and  Pis  used 
for  PBF&B  were  appropriate  for  use  in  two-year  postsecondary  institutions;  and,  if  not, 
(2)  to  develop  appropriate  Pis  through  the  introduction  of  a  predictive  model  of 
community  college  student  performance  which  could  be  used  to  assess  community 
colleges.  The  database  for  this  analysis  contained  longitudinal  data  collected  from  a  -  - 
nationally  representative  sample  of  community  college  students.  The  method  used  to 
address  this  question  was  to  first  determine  the  Pis  most  commonly  used  for  the 
assessment  of  four-year  postsecondary  institutions,  and  to  then  develop  a  model  for 
adapting  those  indicators  to  the  two-year  postsecondary  sector. 

Source  of  Data 

This  study  used  data  from  the  National  Center  for  Education  Statistic 's  (NCES) 
study.  Beginning  Postsecondary  Students  (BPS : 90/94).^  ,  • 

Participants  .  ^ 

Participants  in  the  entire  BPS:90/94  study  were  1 0,624  first-time  beginning 
students  during  the  academic  year  1 989-90,  followed  fi-om  the  1 990  National 
Postsecondary  Student  Aid  Study  (NPSAS:90)  sample.^  Through  the  BPS:90/94  ' 
restricted-use  database,  this  researcher  had  access  to  individual  student  data  for  the  704 
two-year  college  students  in  the  BPS:90/94  sample.  Access  to  the  BPS:90/94  restricted- 
use  data  required  approval  by  the  NCES,  which  was  granted  on  September  30,  1998. 
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Target  Population 

The  target  population  for  this  study  was  all  students  who  entered  postsecondary 
education  at  a  two-year  or  community  college  for  the  first  time  in  academic  year  1 989- 
1990.^ 

BPS:90/94  Sampling  Method 

The  first  step  of  the  BPS  study  (the  NPSAS:90  design)  was  a  stratified  multistage 
sampling  design.^  "The  essence  of  stratification  is  the  classification  of  the  population 
into  subpopulations,  or  strata,  based  on  some  supplementary  information,  and  then  the 
selection  of  separate  samples  from  each  of  the  strata."'^  According  to  Kalton, 
stratification  is  beneficial  because  "sample  sizes  in  the  strata  are  controlled  by  the 
sampler,  rather  than  being  randomly  determined  by  the  sampling  process.  Often  the 
strata  sample  sizes  are  made  proportional  to  the  strata  population  sizes;  in  other  words,  a 
uniform  sampling  fraction  is  used."' '  In  this  case  the  stratified  multistage  sampling 
technique,  as  used  in  the  BPS:90/94  study,  was  non-proportional — the  strata  were  not 
proportional  to  the  strata  population  sizes.  This  does  not  create  a  problem,  however, 
because  the  two-year  sub-population  being  analyzed  in  this  study  (n=704)  was  large 
enough  to  support  valid  and  reliable  analyses.  ,  ,  ■ 

The  use  of  multistage  stratified  sampling  distinguished  this  sample  from  those 
used  in  past  national  longitudinal  studies  of  this  type.  The  target  population  in  this  study 
included  all  first-time  postsecondary  students,  regardless  of  when  they  completed  high 
school.  Because  stratified  sampling  techniques  were  employed,  the  two-year  college  sub- 
sample  (n=704)  was  a  representative  sample  at  the  national  level. For  this  reason,  the 
two-year  and  community  college  portion  of  the  sample  could  thus  be  analyzed  separately 
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from  the  complete  sample  of  10,624  students  in  the  BPS:90/94  working  sample.  "The 
BPS  sample  is  more  likely  than  previous  longitudinal  studies  to  include  some  of  the 
increasing  numbers  of  'nontraditional'  postsecondary  students,  such  as  those  who  have  , 
delayed  their  education  due  to  financial  needs  or  family  responsibilities."'^ 

BPS:90/94  Data  Collection 

The  Beginning  Postsecondary  Students  Longitudinal  Study  (BPS:90/94)  was 
conducted  as  a  follow-up  to  the  1990  National  Postsecondary  Student  Aid  Study 
(NPSAS:90)  conducted  by  the  U.S.  Department  of  Education.  The  NPSAS:90  was 
prompted  by  a  need  for  a  nationally  representative  database  on  postsecondary  student 
financial  aid.  "The  BPS:90/94  followed  students  identified  as  first-time  beginning  (FTB) 
students  in  the  academic  year  1989-90  from  the  NPSAS:90  sample."'"*  A  computer- 
assisted  telephone  interview  (CATI)  obtained  information  concerning  "enrollment, 
program  completion,  education  financing,  employment,  and  family  formation;  graduate 
school  access  and  enrollment;  and  civic  participation."'^ 

Instrumentation 

Computer-Aided  Telephone  Interview  (CATI)  methods  were  used  to  collect  data 
in  the  BPS:90/94  study. '^  The  BPS:90/94  working  sample  included  all  BPS:90/92 
eligible  respondents  and  those  BPS:90/92  non-respondents  for  whom  BPS-eligibility  had 
not  yet  been  determined. ' ^  The  BPS:90/92  locating  database  was  used  to  find  student 
cases  for  the  BPS:90/94  full-scale  survey.  Updated  contact  information  was  obtained 
through  a  national  change  of  address  service.'* 
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The  sampled  students  were  interviewed  on  three  occasions:  in  the  spring  of  1990 
(at  the  end  of  their  first  year  in  college),  spring  of  1992,  and  spring  of  1994.'^  Prior  to 
CATI  operations,  mailings  were  made  to  each  student  in  the  sample.^"  When  CATI 
locating  was  unsuccessful,  "cases  were  sent  to  a  post-CATI  central  trace  for  telephone 
tracing  and,  again,  as  necessary,  field  locating."^' 

Accuracy  of  Estimates 

As  with  all  estimates  derived  from  a  sample,  the  two  broad  types  of  errors  in  this 
study  were  sampling  and  non-sampling  errors.^^  "Random  errors  reduce  the  precision  of 
survey  statistics,  while  non-random  errors  results  in  bias  (i.e.,  estimates  that  do  not 
converge  to  the  true  population  parameter  as  the  sample  size  increases  without  limit)."^^ 

Sampling  errors  occurred  in  this  study  because  the  estimates  were  based  on 
samples  instead  of  the  entire  population.^'*  Non-sampling  errors  are  those  that  occur 
even  in  studies  where  a  complete  census  of  the  population  is  feasible.'^  Sources  of  non- 
sampling  errors  in  this  study  included  the  inability  to  get  complete  information  on 
sample  members,  ambiguously  worded  questions,  interpretation  variability,  the  inability 
or  unwillingness  of  respondents  to  give  accurate  responses,  mistakes  in  recoding  or 
recording  data,  other  errors  in  collection,  and  sampling  and  estimating  of  missing  data.^^ 

According  the  National  Center  for  Education  Statistics  (NCES),  "sampling  errors 
are  primarily  random  errors  for  well-designed  surveys,  like  the  NPSAS  and  BPS."^^ 
The  BPS  instruments  and  data  collection  procedures  underwent  extensive  testing  to 
"minimize  non-sampling  errors  because  these  errors  [were]  difficult  to  quantify  and 
[were]  likely  to  be  non-random  errors."        .     -  .  ,  . 
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Precision  of  the  Survey  Statistics 

The  effect  caused  by  the  accumulation  of  random  errors  in  this  study  was 
measured  by  the  standard  error  of  the  individual  statistics.  A  standard  error  of  a  statistic 
is  the  "estimated  standard  deviation  of  the  sampling  distribution  of  the  statistic  in 

28  * 

repeated  samples  of  the  same  size  using  the  same  sampling  design."    Because  of  this, 
the  standard  error  of  a  survey  statistic  depends  on  the  natural  variability  of  the 
observations  in  the  population,  the  sample  size,  and  the  characteristics  of  the  sampling 
design. 

According  the  National  Center  for  Education  Statistics  (NCES),  "the  features  of 
the  sampling  design  that  affect[ed]  the  sampling  variance  of  the  survey  statistic 
include[ed]  stratification,  multistage  or  cluster  sampling,  and  unequal  probabilities  of 
selection."^^  Of  the  previous  list,  stratification  was  the  only  sampling  design  element  that 
could  have  increased  the  precision  of  the  estimates  (i.e.,  if  the  observations  were 
correlated  with  the  stratification  variables).  The  statistical  adjustment  of  the  analysis 
weights  used  to  reduce  the  potential  for  nonresponse  bias  also  provided  the  opportunity 
for  decreased  precision  of  the  estimates. 

Response  Rates  "  '  - 

The  unweighted  BPS:90/94  response  rate  was  91.4  %  among  those  students 
known  to  be  eligible  for  the  study.^^  For  the  two-year  segment  of  this  sample,  the 
unweighted  response  rates  were  as  follows:  public  institutions  90%;  private,  not-for- 
profit  insfitutions  89.4%;  private,  for-profit  insfitufions  88.6%.  ^' 
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Participant  Summary  Data 

The  NPSAS:90  survey,  the  first  stage  in  the  BPS  study,  provided  information 
concerning  "enrollment,  program  completion,  education  financing,  employment,  and 
family  formation;  graduate  school  access  and  enrollment;  and  civic  participation."^^ 

Research  Design 

Using  the  BPS:90/94  restricted  use  dataset,  this  researcher  selected  variables 
found  by  previous  researchers  to  predict  student  graduation  rates,  retention  rates, 
persistence  rates,  and  other  related  outcomes  of  higher  education.  The  variables  of 
interest  were  then  downloaded  into  an  SPSS  7.5  Professional  Statistics  database  for 
further  manipulation  using  a  Dell  Trinitron  personal  computer  and  SPSS  7.5  Professional 
Statistics  computer  program  for  Windows.  After  the  necessary  recoding  of  variables,  a 
logistic  regression  was  conducted  to  determine  those  variables  most  predictive  of  two- 
year  postsecondary  student  graduation  or  transfer  rates  for  this  sample. 

Statement  of  the  Variables 

Independent  Variables 

Independent  variables  were  those  factors  thought  to  influence  the  outcome 
variable — graduation  or  transfer  from  the  two-year  and  community  college  students' 
first  college  attended.  The  outcome  of  the  interaction  of  the  independent  variables  is 
called  the  dependent  variable.  Each  of  the  ten  independent  variables  selected  for  this 
regression  analysis  was  defined  in  the  BPS:90/94  technical  report,  and  will  be  discussed 
in  the  following  section.^"'       .  , 
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a.  RISK  =  number  of  risk  factors.  This  was  the  number  of  persistence  risk 
factors  possessed  by  an  individual  student.  Risk  factors  included  delayed  enrollment, 
lack  of  high  school  diploma,  part-time  college  attendance,  financial  independence, 
parenthood,  single  parenthood,  and  fiiU-time  employment  during  college. 

b.  ACINT=  index  of  academic  integration  in  1989-90.  This  average 
academic  integration  at  the  NPSAS  institution  examined  whether  an  individual  student 
attended  career-related  lectures,  participated  in  study  groups  with  other  students,  talked 
about  academic  matters  with  faculty,  or  met  with  an  advisor  concerning  academic  plans. 
The  mean  of  the  students'  responses  across  these  questions  was  calculated.  Each 
question  used  in  the  calculation  included  four  possible  responses:  1=  never,  2=  once,  3= 
sometimes,  4=  often.  These  responses  were  then  recoded:  A  student  with  an  average 
academic  integration  less  than  or  equal  to  1.75  was  coded  "low";  a  student  with  an 
average  academic  integration  greater  than  1 .75  but  less  than  or  equal  to  2.75  was  coded 
"moderate";  and  a  student  with  an  average  academic  integration  greater  than  2.75  was 
coded  "high." 

c.  SOCINT=  index  of  social  integration.  The  average  social  integration  at 
the  NPSAS  institution  examined  whether  the  student  had  contact  with  faculty  outside  of 
class,  went  places  with  friends  from  school,  participated  in  student  assistance 
centers/programs,  or  participated  in  school  clubs.  The  mean  of  the  students'  responses 
across  these  questions  was  calculated.  Each  question  used  in  the  calculation  included 
four  possible  responses:  1=  never,  2=  once,  3=  sometimes,  4=  often.  Students  with  an 
average  social  integration  less  than  or  equal  to  1.75  were  coded  "low;"  students  with  an 
average  social  integration  greater  than  1.75  but  less  than  or  equal  to  2.75  were  coded 
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"moderate;"  and  students  with  an  average  social  integration  greater  than  2.75  were  coded 
"high." 

d.  FTWORK=  worked  full  time  while  enrolled.  The  responses  to  this 
question  were:  "did  not  work  full  time  while  enrolled,"  or  "worked  full  time  while 
enrolled."  Students  who  considered  their  primary  role  during  postsecondary  education  as 
employees  were  automatically  considered  to  work  full  time.  The  remaining  students 
were  categorized  as  "working  full  time  while  enrolled,"  if  they  worked  an  average  of 
thirty-four  or  more  hours  per  week. 

e.  EDUCGOAL=  educational  goals  in  1989-90.  This  variable  measured  the 
highest  level  of  education  that  the  student  expected  to  complete,  and  was  recoded  from 
the  student's  reported  educational  aspirations  in  the  NPSAS  survey.  The  categories  for 
the  student  responses  were:  trade  school,  two-year  degree,  bachelor's  degree,  and 
advanced  degree.  This  researcher  assumed  that  a  student  not  interested  in  the  attainment 
of  a  degree  or  certificate  would  respond  "none  of  the  above." 

f.  AGE=  age  in  1 994.  This  was  the  student's  age  as  of  spring  1 994,  derived 
by  adding  five  years  to  the  age  of  the  student  when  he  or  she  began  postsecondary 
education.  The  categories  were:  24  years  or  younger,  25-29  years  old,  30-34  years  old, 
and  35  years  or  older. 

g.  PARED=  parental  education.  Parental  level  of  education  was  defined  as 
the  highest  level  of  education  completed  by  either  parent.  The  categories  were:  less  than 
high  school  diploma;  high  school  graduate;  some  postsecondary  education;  and 
bachelor's  degree  or  higher. 
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h.  RELIGION=  student's  religion.  Because  past  research  indicated  that  both 
Catholic  and  Jewish  religions  were  positively  correlated  with  success,^'*  this  variable  was 
recoded  to  indicate  whether  the  student  was  Catholic,  Jewish,  or  neither. 

i.  NUMDEP=  student's  number  of  dependents.  This  variable  reflected  the 
number  of  dependents  the  student  claimed  on  his  or  her  tax  return  in  1989-90. 

j.        ABILITYH=  ability  index  high.  This  variable  was  an  indicator  of  a 
student's  perception  of  his  or  her  past  academic  performance,  relative  to  other  students. 
Students  with  high  self-perception  of  their  past  academic  performance,  relative  to  other 
students,  scored  high  on  this  variable. 

Moderator  Variables 

Moderator  variables  were  those  variables  used  to  select  a  certain  group  from 
within  the  larger  sample — ^in  this  case,  beginning  two-year  postsecondary  students.  The 
moderator  variables  used  for  this  study  were: 

a.       LEVELCON^  level  and  control  of  institution.  Level  and  control  of  the 
referent  institution  was  an  aggregate  of  two  variables:  (1)  level  of  institution,  and  (2) 
control  of  institution.  Of  interest  to  this  researcher  was  the  group  coded  "4"-^ublic  and 
private  two-year  institutions.  A  public  institution  was  defined  as  "a  postsecondary 
educational  institution  operated  by  publicly  elected  or  appointed  school  officials  in  which 
the  program  and  activities  were  under  the  control  of  these  officials  and  was  supported 
primarily  by  public  fimds."^'^  A  private,  not-for-profit  institution  was  defined  as  "a 
postsecondary  educational  institution  controlled  by  an  independent  governing  board  and 
incorporated  under  Section  501(c)(3)  of  the  Internal  Revenue  Code."^^  A  two-year 
mstitution  was  defined  as  "an  institution  whose  program  of  study  result[ed]  in  an  award 
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or  degree  below  the  baccalaureate  level,  and  [was]  at  least  [two]  years  but  less  than  [four] 
years  in  duration.  These  institutions  [included]  many  community  and  junior  colleges."''^ 

b.       INSTID=  institution  I.D.  Each  institution  was  assigned  a  unique 
identifier.  This  variable  reflected  those  assignments. 

Dependent  Variable 

The  dependent  variable  is  the  outcome  variable  of  interest — ^in  this  case, 
graduation  or  transfer  from  the  first  postsecondary  institution  attended.  The  following  is 
a  descripfion  of  the  outcome  variable  ufilized  in  the  logisfic  regression: 

OUTCOME=  the  student  outcome  of  interest  to  this  researcher  was  graduation  or 
transfer  from  a  two-year  institution  within  the  five  years  fi-om  1990  to  1994  in  which  the 
BPS:90/94  study  was  conducted.  Because  this  variable  was  a  record  of  whether  the 
student  graduated  or  transferred  fi-om  any  postsecondary  institution  (not  necessarily 
his/her  beginning  institution),  this  outcome  variable  measured  student  graduation  and 
transfer  in  their  broadest  sense. 

Method  of  Collection 

The  data  utilized  in  this  study  were  obtained  from  the  NCES  through  a  process  of 
verification  and  permission  completed  on  September  30,  1998.  ; 

Individual  student  data  concerning  each  of  the  variables  of  interest  were  selected 
and  placed  in  an  SPSS  7.5  Professional  Statistics  data  file  for  fiirther  manipulafion  and 
analyses.  More  information  about  that  process  is  located  in  the  next  section,  entitled 
"Method  of  Analysis."  >     -    :  .  .  .  , 
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Method  of  Analysis 

Independent  variables,  moderator  variables,  and  dependent  variables  were 
selected  for  inclusion  in  the  logistic  regression  analyses  from  the  Electronic  Code  Book 
(ECB)  included  with  the  BPS:90/94  restricted  use  dataset  obtained  from  NCES.  These 
variables  were  then  downloaded  into  an  SPSS  7.5  Professional  Statistics  file  for  ftirther 
manipulation.  This  working  data  base,  an  SPSS  7.5  Professional  Statistics  file  containing 
only  the  variables  of  interest  to  this  researcher,  were  then  filtered  to  include  only  those 
students  whose  first  postsecondary  institution  was  a  two-year  college.  Prior  to  initiation 
of  any  statistical  analysis,  all  data  were  verified  for  coding  accuracy.  Analyses  were 
performed  using  SPSS  7.5  Professional  Statistics  for  Windows. 

,  After  selecting  variables  to  be  included  in  the  regression  analysis,  it  was 
necessary  to  recode  some  of  the  variables.  For  instance,  the  student  outcome  variable 
was  receded  as  a  dichotomous  dummy  variable  to  reflect  two  categories:  "attain/transfer" 
(yes=l  or  no=0). 

According  to  SPSS  Inc.,  all  things  being  equal,  logistic  regression  requires  larger 
sample  sizes  than  does  Ordinary  Least  Squares  multiple  regression.^^  Because  of  this 
fact,  it  was  appropriate  to  replace  all  "system  missing"  data  with  that  variable's  mean 
(i.e.,  the  average  score  for  that  variable)  to  avoid  the  exclusion  of  cases.  The  recoding  of 
missing  data  in  this  way  allowed  the  two-year  college  sample  size  to  remain  relatively 
large  (this  sample  size  was  704).  This  technique  is  commonly  used  in  situations  such  as 
this,  where  the  number  of  system  missing  data  represents  only  a  small  portion  of  the  total 
variables.  The  following  variables  required  recoding  to  account  for  missing  data:  the 
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ability  index  variable,  the  social  and  academic  integration  variables,  age,  number  of 
dependents,  parents'  education  level,  and  the  risk  variable. 

Frequencies  were  run  for  all  variables  to  check  for  any  inconsistencies  in  the  data. 

Determination  of  the  Model 

To  identify  any  inconsistencies  or  errors  in  the  data,  descriptive  statistics  were 
produced  for  all  variables  in  the  working  dataset.  It  was  determined  that  logistic 
regression  was  appropriate  for  this  data  due  to  the  dichotomous  and  categorical  nature  of 
the  dependent  variable.  In  this  case,  the  dependent  variable  was  not  the  raw  data  values, 
but  rather,  the  odds  of  the  event  of  interest  occurring  (graduation  or  transfer  from  a  two- 
year  postsecondary  institution  within  the  five  years  of  this  study). 

Because  logistic  regression  analysis  was  designed  to  use  a  mix  of  continuous  and 
categorical  predictor  variables  to  predict  a  categorical  outcome,  or  dependent,  variable,  it 
was  used  for  these  analyses.  An  in-depth  explanation  for  the  use  of  logistic  regression 
analyses  can  be  found  in  the  next  section,  "Selection  of  the  Model."   ■  •■  • 

Selection  of  the  Model 

"Multiple  regression  cannot  be  used  to  predict  the  values  of  a  dichotomous 
dependent  variable  for  two  reasons.  First,  when  [used]  to  predict  the  values  of  a  variable 
coded  ...  0  or  1  we  ...  consider[ed]  the  predicted  values  to  be  probabilities,  that  is,  the 
probability  of  obtaining  a  predicted  value  of  1 .  In  multiple  regression  with  a  straight-line 
fit  to  the  data,  it  [was]  often  the  case  that  values  less  than  0  or  greater  than  1  [were] 
predicted.  That  [was]  clearly  a  problem."-^^ 


90 

There  were  several  reasons  to  use  logistic  regression — ^as  opposed  to  multiple 
regression  analysis  or  discriminant  analysis — ^for  this  research.  First,  logistic  regression 
was  most  appropriate  for  this  data  because  "the  assumptions  necessary  for  hypothesis 
testing  in  regression  analysis  [were]  necessarily  violated."^°  Predicted  values  derived 
from  a  multiple  regression  analysis  could  not  be  interpreted  as  probabilities  because  they 
were  not  "constrained  to  fall  in  the  interval  between  0  and  1       In  turn,  since  logistic 
regression  analysis  required  "far  fewer  assumptions  than  discriminant  analysis,"^^  it  was 
thought  that  logistic  regression  would  perform  better  than  the  aforementioned  analysis 
techniques.  .  • 

Logistic  regression  solved  some  of  the  problems  involved  with  using  multiple 
regression  analyses  for  categorical  dependent  variables.  Developed  in  the  1 960s,  logistic 
regression  allowed  this  researcher  to  predict  the  value  of  a  variable  that  varied  on  a  scale 
from  zero  to  one.  The  logistic  regression  curve  is  S-shaped,  and  therefore  accounted  for 
the  non-linear  nature  of  the  relationship  between  the  independent  and  dependent  variables 
utilized  in  this  research. 

Subsequent  to  the  determination  of  the  functional  form,  the  model. 
In  (Odds)  =  a  +  pi  Xi  +  p2  X2  +  . . .+  pk  Xk,  was  selected  for  use  in  ftirther  analyses.  The 
variables  on  the  right  side  of  the  equation  were  representations  of  the  independent 
variables  and  the  intercept  in  a  regression  equation.  Those  on  the  left  side  of  the  equation 
were  the  natural  log  of  the  odds;  the  quantity  In  (Odds)  was  called  a  logit. 

The  odds  were  related  to  the  probability  by: 

Odds  =  Probability 
1 -Probability 
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It  is  important  to  note  that  "there  is  a  linear  relationship  with  the  independent 
variables  in  logistic  regression,  but  it  is  linear  in  the  log  odds  and  not  in  the  original 
probabilities."^^ 

The  logistic  equation  was  transformed  into  an  equation  in  the  probability,  which 
follows: 

Prob  (event)  =  e  "^P,^,  ^P/^-^PA 

1 +^        II      2  2         i  i 

To  analyze  this  equation  the  maximum-likelihood  method  was  employed.  The 
maximum-likelihood  method  derived  coefficients  that  made  observed  values  most  likely 
for  the  given  set  of  independent  variables.  Of  the  two  general  goals  for  logistic 
regression — determining  the  effect  of  a  set  of  variables  on  the  probability  of  an  event 
occurring,  and  attaining  the  highest  predictive  accuracy  possible  with  a  given  set  of 
independent  variables — the  latter  was  most  important  to  this  researcher. 

Once  the  selection  of  a  set  of  independent  variables  (i.e.,  predictor  variables)  was 
completed  and  recoded,  the  application  of  a  filter  for  the  level  and  control  of  the 
institution  (where  "4"  represented  the  students  attending  two-year  institutions)  was 
employed,  and  three  types  of  regression  were  run. 

The  step-wise  method  of  model  selection  was  employed  to  find  the  best  subset  of 
independent  variables  to  predict  the  dependent  variable.  Using  SPSS  7.5  Professional 
Statistics  software,  a  forward  step-wise  WALD  was  run  at  the  .05  level  of  significance. 

There  are  two  possible  purposes  for  using  logistic  regression  to  analyze  data:  (1) 
to  "determine  the  effect  of  a  set  of  variables  on  the  probability  of  an  outcome,  plus  the 
effect  of  individual  variables,"^  or  (2)  to  maximize  the  accuracy  of  predicting  the 
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probability  of  the  outcome  variable  utilizing  a  set  of  predictor  variables.  According  to 
SPSS  Inc.,  these  two  functions  of  logistic  regression  operate  individually  from  one 
another."*'  Because  this  researcher  was  most  interested  in  "predicting  whether  a  future 
event  will  fall  in  one  or  the  other  category  of  the  dependent  variable"  it  was  appropriate 
to  focus  on  the  second  goal  of  logistic  regression.''^ 

The  Classification  Table  produced  by  logistic  regression  analysis  is  a  measure  of 
how  well  the  model  performs  because  it  is  a  depiction  of  how  accurately  cases  are 
classified  into  the  two  categories  of  the  dependent  variable  (i.e.,  whether  or  not  a  student 
graduated  or  transferred  from  a  two-year  college  within  the  five  years  of  the  BPS:90/94 
Study).  In  other  words,  the  Classification  Table  addresses  how  much  the  prediction  is 
improved  over  chance.'*^ 

An  appropriate  technique  for  maximizing  the  predictive  accuracy  of  a  logistic 
regression  model  includes  the  manipulation  of  the  cut-off  point  used  for  the  Classification 
Table."*^  Because  of  this,  it  was  appropriate  to  test  several  cut-off  points  for  the 
Classification  Table  in  order  to  maximize  the  predictive  power  of  the  developed  model. 

An  examination  of  the  probability  of  the  observed  results,  given  the  parameter 
estimates  (known  as  the  log  of  the  likelihood),  was  conducted  and  used  as  a  measure  of 
the  model's  fit  to  the  data. 

The  Nagelkerke  Pseudo     was  examined  to  estimate  the  amount  of  variance  in 
the  dependent  variable  that  could  be  accounted  for  by  the  independent  variables  in  the 
model. 

To  check  the  significance  of  the  independent  variables  to  the  model,  the  Wald 
statistic  was  calculated.  Multicollinearity  was  checked  to  examine  the  relationships 
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between  the  independent  variables.  The  resuhs  of  the  aforementioned  data  are  included 
in  Chapter  4. 

Means 

Means  are  averages  of  the  aggregate,  similar  data  (e.g.,  the  average  student  age  of 
the  sample  population).  Data  means,  as  appropriate,  were  calculated  and  are  reported  in 
Chapter  4.  Means  were  calculated  for  each  independent  variable. 

Model  Chi-square 

.  The  Model  Chi-square  is  a  statistical  test  of  the  null  hypothesis  that  the 
coefficients  for  all  of  the  terms  in  the  model  are  zero.  This  test  was  conducted  and  is 
equivalent  to  the  overall  F  test  in  regression.  The  Model  Chi-square  is  the  difference 
between  the  initial  and  final  log  likelihood.  Like  the  overall  F  test  in  regression,  the 
Model  Chi-square  tests  the  overall  variation  in  the  dependent  variable  as  influenced  by 
the  independent  variables.  Results  of  the  Model  Chi-square  are  reported  in  Chapter  4. 

Summary 

This  chapter  began  with  an  introductory  discussion  of  the  nature  of  the  problem  to 
be  examined  in  this  research  study,  and  included  a  discussion  of  the  target  population  and 
the  design  of  the  study. 

The  independent  and  dependent  variables  used  in  the  analyses  were  identified  and 
defined.  The  sources  of  data,  methods  of  collection,  model  determination,  selection  of 
the  model,  and  methods  of  analysis  were  also  discussed. 
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The  results  of  the  study,  including  presentation  of  the  data  and  analyses,  can  be 
found  in  Chapter  4.  A  summary  of  the  findings  is  included  at  the  end  of  that  chapter. 
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CHAPTER  4 
RESULTS 


Introduction 

The  purpose  of  this  study  was  to  examine  the  use  of  performance  assessment  and 
performance-based  funding  and  budgeting  (PBF&B)  in  United  States  community 
colleges.  This  study  was  designed  (1)  to  determine  whether  current  models  and 
performance  indicators  (Pis)  used  for  PBF&B  were  appropriate  for  assessing  the 
performance  of  two-year  postsecondaiy  institutions;  and,  if  not,  (2)  to  develop 
appropriate  Pis  through  the  introduction  of  a  predictive  model  of  community  college 
student  performance  which  could  be  used  to  assess  community  colleges. 

In  order  to  address  some  of  these  issues,  this  research  examined  the  following  two 
areas  of  inquiry: 

The  first  question  was  determining  how  performance  assessment  and  performance 
measures  were  used  to  allocate  state  funds  to  postsecondary  education  institutions.  Was 
performance  assessment  used  for  all  types  of  postsecondary  institutions— both  two-year 
and  four-year  colleges  and  universities?  Which  states  had  used,  were  using,  and/or 
planned  to  use  this  type  of  funding  allocation?  Of  the  states  currently  using  performance 
funding  or  budgeting,  what  portions  of  their  individual  budgets  did  they  allocate  in  this 
manner?  Did  states  use  multiple  measures  of  institutional  performance  to  allocate  funds, 
and,  if  so,  how  many  did  they  use?  Were  institutional  performance  measures  the  same 


97 


98 

for  all  postsecondary  institutions?  Specifically,  was  performance  measured  differently 
for  four-year  and  two-year  colleges  and  universities?  What  measures  of  postsecondary 
institutional  performance  did  states  most  frequently  use  to  allocate  funds? 

This  research  question  was  answered  through  a  review  of  the  literature  and  was 
addressed  in  Chapter  2. 

The  second  question  was  whether  the  institutional  performance  measures  used  by 
states  and  institutions  accurately  measured  the  performance  of  community  and  two-year 
colleges.  Could  the  outcome  measures  most  commonly  used  by  states  and  institutions 
accurately  depict  community  and  two-year  college  performance?  How  feasible  was  the 
development  of  performance  measures  to  account  for  differences  in  levels  and  types  of 
institutions?  Specifically,  was  there  a  single  model  that  could  serve  as  an  accurate 
performance  indicator  for  both  two-year  and  four-year  postsecondary  institutions? 

This  research  question  was  answered  through  a  review  of  the  literature  and 
through  the  use  of  logistic  regression  analysis  to  develop  a  model  of  commimity  college 
student  graduation  and  transfer  rates  and,  subsequent  two-year  college  performance.  This 
chapter  will  include  the  results  of  the  logistic  regression  analyses,  which  addressed  the 
second  research  question. 

Variables 

There  were  ten  major  determinants  of  two-year  and  community  college  student 
graduation  and  transfer  rates  upon  which  there  appear  to  be  general  agreement  among  the 
postsecondary  education  research  community.  The  following  were  the  independent 
variables  utilized  in  this  study:  the  number  of  known  risk  factors  that  the  student 
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possessed;  the  student's  index  of  academic  integration;  the  student's  index  of  social 
integration;  whether  or  not  the  student  worked  full-time  while  enrolled;  the  educational 
aspirations  of  the  student;  the  age  of  the  student;  the  student's  parents'  education;  the 
student's  religion;  the  student's  number  of  dependents;  and  the  student's  self-reported 
ability  level.  The  graduation  or  the  transfer  of  the  student  from  the  two-year  or 
community  college  was  the  dependent  variable  utilized  for  this  study. 

Independent  Variables 

.    The  definitions  of  the  selected  independent  variables  included  in  this  study  were: 

a.  The  number  of  risk  factors,  as  measured  by  the  RISK  variable,  was  the 
number  of  persistence  risk  factors  possessed  by  an  individual  student. 
Risk  factors  included  delayed  enrollment,  lack  of  high  school  diploma, 

part-time  college  attendance,  financial  independence,  parenthood,  single 
parenthood,  and  full-time  employment  during  college. 

b.  The  index  of  academic  integration  in  1 989-90,  ACINT,  was  the  average 
academic  integration  of  an  individual  student  at  the  NPSAS  institution 

i      first  studied.  This  variable  measured  whether  or  not  a  student  attended 
career-related  lectures,  participated  in  study  groups  with  other  students, 
talked  about  academic  matters  with  faculty,  or  met  with  an  advisor 
concerning  academic  plans. 

c.  The  social  integration  index,  SOCINT,  was  the  student's  average  social 
integration  at  the  first  institution  s/he  attended.  This  variable  examined 
whether  the  student  had  contact  with  faculty  outside  of  class,  went  places 


100 

with  friends  from  school,  participated  in  student  assistance 
centers/programs,  or  participated  in  school  clubs. 
Whether  or  not  a  student  worked  fiill  time  while  enrolled  at  the  first 
postsecondary  institution  was  measured  by  the  FTWORK  variable. 
A  student's  educational  aspirations  in  1989-90  were  measured  by  the 
EDUCGOAL  variable.  The  categories  for  the  student  responses  were: 
trade  school,  two-year  degree,  bachelor's  degree,  and  advanced  degree. 
A  student's  age  in  1994,  measured  by  the  AGE  variable,  was  derived  by 
adding  five  years  to  the  student's  age  at  the  beginning  of  their 
postsecondary  education  career. 

The  parental  level  of  education  of  a  student  was  defined  as  the  highest 
level  of  education  completed  by  either  parent  and  was  measured  by  the 
PARED  variable. 

A  student's  religion,  measured  by  the  RELIGION  variable,  was  recoded  to 
indicate  whether  the  student  was  Catholic,  Jewish,  or  neither. 
A  student's  number  of  dependents,  measured  by  the  NUMDEP  variable, 
reflected  the  number  of  dependents  the  student  claimed  on  his  or  her  tax 
return  in  1989-90. 

A  student's  ability  index,  measured  by  the  ABILIYH  variable,  was  an 
indicator  of  a  student's  perception  of  his  or  her  past  academic 
performance,  relative  to  other  students. 
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Moderator  Variable 

Moderator  variables  are  those  variables  used  to  select  out  a  sub-group  from  the 
larger  sample — in  this  case,  begimiing  two-year  postsecondary  students  were  a  sub-group 
of  the  larger  sample  that  contained  a  nationally  representative  sample  of  all  beginning 
postsecondary  students  from  all  types  of  postsecondary  institutions.  The  moderator 
variable  used  for  this  study  was  the  level  and  control  of  the  institution. 

The  level  and  control  of  the  institution  a  student  attended,  indicated  by  the 
LEVELCON  variable,  was  the  variable  used  to  focus  this  research  on  two-year  public 
college  students  in  the  larger  sample  of  all  beginning  postsecondary  students.  This 
variable  was  an  aggregate  of  two  variables:  (1)  level  of  institution,  and  (2)  control  of 
institution.  An  institution's  level  was  defined  by  the  highest  degree  offered  by  the 
institution  (e.g.,  associate's,  bachelor's,  master's,  doctorate).  The  type  of  institutional 
funding  and  governance  defined  an  institution's  control  (e.g.,  public,  private,  proprietary, 
nonprofit). 

Dependent  Variable 

The  dependent  variable  was  the  outcome  variable  of  interest  to  this  researcher— in 
this  case,  graduation  or  transfer  fi-om  a  two-year  postsecondary  institution.  The  following 
is  a  description  of  the  outcome  variable  utilized  in  the  logistic  regression. 

The  OUTCOME  variable  was  the  dependent  variable  predicted  by  the 
independent  variables  listed  above.  The  OUTCOME  variable  measured  whether  or  not  a 
student  graduated  with  a  college  degree  or  transferred  from  a  two-year  institution  within 
the  five  years  from  1990  to  1994  in  which  the  BPS:90/94  study  was  conducted.  Because 
this  variable  was  a  record  of  whether  the  student  graduated  or  transferred  from  any 
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postsecondary  institution  (not  necessarily  his  or  her  beginning  institution),  this  outcome 
variable  measured  student  graduation  and  transfer  behavior  in  the  broadest  sense. 

Descriptive  Statistics 

Range  of  the  Data 

The  reported  values  for  each  of  the  variables  included  in  this  study  may  be 
expressed  as  a  range  (the  difference  between  the  lowest  value  and  the  highest  value). 
Table  4-1  includes  the  ranges  for  each  variable  included  in  the  logistic  regression 
analyses. 

Means 

A  mean  is  an  arithmetic  average  of  observations.  Table  4-1  includes  the  means  for 
each  of  the  variables  included  in  the  logistic  regression  analyses. 

Standard  Deviations  and  Frequencies 

The  standard  deviation  of  a  variable  is  a  "measure  of  the  spread  of  data  about 
their  mean  and  an  essential  part  of  many  statistical  tests."'  The  frequency  of  a  variable  is 
a  tally  that  takes  the  form  of  either  numbers  or  percentages.  Table  4-1  includes  standard 
deviations  and  frequencies  (expressed  as  numbers)  of  the  independent  variables  utilized 
in  this  regression  analysis. 

ABILITYH  had  a  frequency  of  704  (i.e.,  there  were  704  student  who  responded  to 
this  question).  The  range  for  ABILITYH  was  10  with  a  minimum  of  0  and  maximum  of 
1 0.  In  this  case,  1 0  represented  the  student  who  felt  she  was  very  much  above  her 
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average  peer  academically.  ABILITYH  had  a  mean  of  1 .04  with  a  standard  deviation  of 
1 .2 1  and  a  variance  of  1 .47. 


Table  4-1.  Descriptive  Statistics. 


RISK 

ACINT 

SOCINT 

FTWORK 

EDUCGOAL 

AGE 

PARED 

RELIGION 

NUMDE 

ABILITYH 

Mean 

Mean 

Mean 

Mean 

Mean 

Mean 

Mean 

Mean 

Mean 

Mean 

1.91 

2.37 

2.05 

.23 

.88 

22.31 

3.59 

.31 

.37 

1.04 

Median 

Median 

Median 

Median 

Median 

Median 

Median 

Median 

Median 

Median 

1.00 

2.00 

2.00 

0 

0 

19.00 

3.00 

0 

0 

1.00 

Mode 

IVIUUC 

IVIUUC 

iviuue 

ivioae 

ivioue 

ivioae 

0 

2 

2 

0 

1 

18 

2 

0 

0 

0 

Range 

Range 

Range 

Range 

Range 

Range 

Range 

Range 

Range 

Range 

7 

3 

3 

1 

1 

52 

6 

1 

6 

10 

Min.  0 

Min.  1 

Min.  1 

Min.  0 

Min.  0 

Min.  16 

Min.  1 

Min.  0 

Min.-l 

Min.  0 

Max.  7 

Max.  4 

Max.  4 

Max.  1 

Max.  1 

Max. 
68 

Max.  7 

Max.  1 

Max.  5 

Max. 
10 

SD 

SD 

SD 

SD 

SD 

SD 

SD 

SD 

SD 

SD 

1.76 

.80 

.73 

.47 

.33 

7.71 

2.00 

.46 

.86 

1.21 

FREQ 
704 

FREQ 
704 

FREQ 
704 

FREQ 
672 

FREQ 
704 

FREQ 
704 

FREQ 
704 

FREQ 
704 

FREQ 
704 

FREQ 
704 

FTWORK  was  a  dichotomous  variable  with  0  representing  a  student  who  did  not 
work  full  time  while  a  1  represented  a  student  who  did.  Of  this  group,  67.6%  reported 
that  they  did  not  work  fiill-time  while  enrolled  and  32.4%  reported  working  full  time 
while  enrolled. 
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ACINT  measured  the  student's  perception  of  her  academic  integration  at  the 
college  she  attended.  This  four-point  scale  ranged  from  1  to  4  with  4  representing  the 
highest  degree  of  academic  integration.  The  most  common  responses  on  this  four-point 
scale  were  2  (41.3%)  and  3  (36.5%),  which  represented  a  student  who  participated  in  2  or 
3  of  4  possible  activities  that  represented  academic  integration.  The  mean  score  for  this 
variable  was  2.37,  the  standard  deviation  was  .80,  and  a  plot  of  this  variable's  frequencies 
approximated  a  normal  curve.  •  . 

SOCINT  represented  the  student's  perception  of  his  social  integration  at  the 
college  he  attended.  Of  the  704  students  responding  to  this  four-point  question,  the  mean 
was  2.05,  with  a  standard  deviation  of  .73 .  The  plot  of  this  variable's  frequencies 
showed  a  positive  skewness  of  .428 — more  than  half  of  the  respondents  (54.7%) 
participated  in  two  of  the  four  activities  associated  with  social  integration. 

PARED,  which  represented  the  education  level  of  students'  parents,  resulted  in  a 
mean  of  3.59  (they  had  completed  a  trade  school  program  but  had  less  than  2  years  of 
college).  The  mode  for  this  group  (2.0)  showed  that  3  5 .2%  of  the  students  had  parents 
whose  highest  level  of  education  was  high  school  completion  with  another  12.8%  whose 
parents  had  less  than  a  high  school  diploma.  These  results  indicated  that  48.0%  of  the 
two-year  college  student  subsample  were  first  generation  college  students. 

/  The  RISK  variable  was  a  constructed  variable  with  a  range  of  7  (from  zero  to  7). 
This  variable  was  a  count  of  the  number  of  factors  a  student  possessed  that  were  known 
*  to  be  risks  to  student  completion  of  a  college  degree.  With  704  responses,  this  variable  ; 
had  a  mean  of  1 .91  and  a  mode  of  0  (29.5%  of  the  respondents  indicated  that  they  did  not 
possess  any  risk  factors).  This  variable's  skewness  was  .566,  kurtosis  was  -.814,  and 
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standard  deviation  was  1.76.  Approximately  35%  of  the  students  possessed  3  or  more 
risk  factors. 

The  AGE  variable  indicated  whether  or  not  a  student  had  delayed  her  college 
attendance  after  high  school  graduation — a  known  risk  factor  for  student  graduation  and 
transfer  behavior.  41 .2%  of  the  students  were  the  traditional  college  freshman  age  of  18- 
years-old.  1 7.8%  of  the  respondents  appeared  to  have  delayed  their  college  enrollment 
by  one  year.  3 1 .8%  of  the  respondents  were  20  years  old  or  above  at  the  time  of  their 
first  enrollment  in  college,  which  indicated  that  this  group  might  have  delayed  their 
college  enrollment  after  high  school  completion  by  two  or  more  years. 

According  to  the  NUMDEP  variable,  78.6%  of  the  704  student  respondents 
indicated  that  they  did  not  have  dependents  other  than  a  spouse.  21 .4%  had  one  or  more 
dependents  (other  than  a  spouse). 

The  RELIGION  variable  indicated  that  29.5%  of  this  subsample  indicated  that 
they  were  Catholic  or  Jewish  and  the  remaining  70.5%  were  not. 

The  educational  aspirations  of  the  students  sampled,  EDUCGOAL,  indicated  that 
87.5%  of  the  students  in  the  two-year  college  subsample  were  enrolled  with  the  goal  of 
completing  either  a  bachelor's  or  associate's  degree. 

The  dependent  variable  of  interest,  OUTCOME,  indicated  that  55.4%  of  the  two- 
year  subsample  did  not  transfer  or  graduate  from  a  postsecondary  institution  within  the 
five  years  of  this  study.  The  remaining  44.6%  did  graduate  or  transfer  within  the  five- 
year  period  of  this  study. 
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Analysis  of  the  Data 

The  data  utilized  in  this  study  were  examined  for  the  purpose  of  selecting  the 
independent  variables  and  functional  form  that  best  predicted  the  graduation  and  transfer 
rates  of  two-year  and  community  college  students  in  the  BPS:90/94  study.^ 

Preliminary  analysis  of  the  data  (plotting)  and  this  researcher's  knowledge  of 
dichotomous  dependent  variables  indicated  that  the  data  was  of  a  non-linear  fiinctional 
form.  Thus,  the  equation  developed  for  subsequent  analyses  was  of  the  non-linear  form. 
A  logistic  regression  model  was  developed  to  explain  the  relationship  between  the 
independent  variables  and  the  dependent  variable. 

To  hypothesize  the  expected  signs  of  the  correlation  coefficients,  the  following 
variables  were  included  with  their  anticipated  corresponding  signs: 


GRADUATION/TRANSFER  =/(RISK,  ACINT,  SOCINT,  FTWORK,  EDUCGOAL, 

-  +  +  _  + 

AGE,  PARED,  RACE,  RELIGION,  NUMDEP,  ABILITY) 

=  Po  -  Pi  RISK  +  P2ACINT  +  Pj  SOCINT  -  P4  FTWORK+  P5EDUCGOAL  - 
PsAGE  +  pvPARED  +  Pg  RACE  +  P9RELIGON-  P9NUMDEP  +  p,oABILITY  +  e, 


Because  the  literature  indicated  that  increases  in  the  number  of  risk  factors  a 
student  possessed  would  decrease  his  likelihood  of  completing  a  college  degree,  it  was 
assumed  in  this  research  that  the  RISK  variable  would  be  negatively  correlated  with  the  ' 
OUTCOME  variable.  Similarly,  the  degree  to  which  a  student  was  academically 
integrated  with  the  institution  she  attended  was  assumed  to  be  positively  correlated  with  ' 
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the  OUTCOME  variable  of  interest  in  this  research.  In  the  literature,  it  was  found  that  a 
student's  social  integration  would  increase  the  probability  of  a  student  graduating  from  a 
postsecondary  institution.^  Therefore,  it  was  assumed  that  the  SOCINT  variable  would 
be  positively  correlated  with  the  dependent  variable  (graduation  or  transfer  from  a  two- 
year  college  within  the  five  years  of  the  BPS:90/94  study). 

The  FTWORK  variable,  which  measured  whether  or  not  a  student  worked  fiill 
time,  was  assumed  to  have  a  negative  relationship  with  the  OUTCOME  variable  because 
a  student  with  a  ftill-time  job  was  believed  to  have  less  time  to  concentrate  on  college 
studies. 

A  student's  educational  goals,  as  measured  by  the  EDUCGOAL  variable,  was 
believed  to  have  a  positive  relationship  with  the  OUTCOME  variable.  The  higher 
numbers  on  the  scale  represented  the  higher  educational  aspirations  of  a  student — a 
student  who  did  not  intend  to  graduate  or  transfer  would  be  less  likely  to  do  so. 

The  AGE  of  a  student  was  assumed  to  have  a  negative  relationship  with  the 
OUTCOME  variable  because  prior  research  found  that  the  longer  a  student  delayed 
college  enrollment  after  high  school  graduation,  the  less  likely  that  student  was  to 
continue  on  to  college  graduation. 

The  level  of  parental  education,  measured  by  the  PARED  variable,  was  assumed 
to  have  a  negative  relationship  with  the  OUTCOME  variable  because  prior  research 
indicated  that  students  coming  from  households  where  neither  parent  had  attended 
college  were  less  likely  to  graduate  from  college. 

A  student's  religion,  as  indicated  by  the  RELIGION  variable,  was  assumed  to 
have  a  positive  relationship  with  the  OUTCOME  variable  because  prior  research 
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indicated  that  Catholic  and  Jewish  students  were  more  likely  to  graduate  from  college 
than  were  non-Catholic  and  non- Jewish  students. 

The  number  of  financial  dependents  a  student  possessed  was  assumed  to  have  a 
negative  relationship  with  the  OUTCOME  variable  because  it  was  believed  that  the  more 
people  for  which  a  student  was  financially  responsible,  the  less  time  the  student  would 
have  to  devote  to  college  studies. 

The  student's  perception  of  his  academic  abilities,  ABILITYH,  was  assumed  to 
have  a  positive  relationship  with  the  OUTCOME  variable.  This  assumption  was  based 
on  prior  research  that  found  that  a  student's  high  school  grade  point  average  (GPA)  was 
highly  predictive  of  a  student's  likelihood  of  graduation  from  college.  Because  the 
ABILITYH  variable  was  seen  as  a  proxy  for  a  student's  GPA  it  was  assumed  that  it,  too, 
would  have  a  strong  positive  relationship  with  a  student's  likelihood  to  graduate  fi-om 

Missing  Data  '  ^  ^ 

Although  the  sample  size  was  704  for  the  two-year  college  subsample,  some 
variables  contained  missing  data,  which  resulted  in  a  somewhat  lower  sample  size. 
Overall,  the  missing  data  resulted  in  no  less  than  615  responses  for  each  variable  included 
in  the  analysis.  Once  the  logistic  regression  analysis  omitted  variables  from  the  final 
model  there  were  30  rejected  cases  due  to  missing  data— the  resulting  sample  size  was 
674.  According  to  SPSS  Inc.,  "all  things  being  equal,  logistic  regression  requires  larger 
sample  sizes  than  does  [Ordinary  Least  Squares]  regression  for  the  correct  inference  ... 
[ideally  there  should  be]  at  least  thirty  times  as  many  cases  as  parameters  being  estimated 
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in  the  model."''  Using  the  aforementioned  rule,  the  sample  size  for  this  research  met  all 
such  requirements. 

Correlations 

The  results  of  bivariate  correlations  showed  that  29  correlations  were  statistically 
significant  at  the  .05  alpha-level. 

Table  4-2  displays  the  correlations  that  were  significant  at  the  .05  alpha-level. 

■  "fS, 

Purpose  of  the  Model 

To  address  the  final  research  question  pertaining  to  the  creation  of  a  flexible 
institutional  performance  measure,  a  logistic  regression  analysis  was  conducted  to 
identify  a  model  that  could  accurately  predict  community  college  student  graduation  and 
transfer  behavior  at  two-year  institutions.  The  use  of  logistic  regression  analysis  to 
develop  a  model  that  could  accurately  predict  two-year  college  student  graduation  and 
transfer  behavior  was  an  important  step  toward  developing  a  flexible  performance 
measure  for  postsecondary  institutions.  The  logistic  regression  model  could  offer  a  way 
to  account  for  the  student  characteristics  that  were  largely  out  of  institutional  control,  but 
which  contributed  to  the  likelihood  of  college  student  graduation  or  transfer  within  five 
years  (e.g.,  student's  educational  aspirations,  parental  level  of  education,  etc.). 

An  accurate  regression  model  could  utilize  incoming  student  characteristics  to 
predict  college  student  outcome— a  figure  that  could  later  be  compared  to  actual  student 
graduation  and  transfer  behavior.  The  residual  (predicted  versus  actual  student 
graduation  and  transfer  rates)  could  offer  a  fair  comparison  of  institutional  performance 
across  colleges  and  universities.  This  analysis  identified  the  variables  most  predictive  of 
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student  graduation  and  transfer  behavior  by  eliminating  the  independent  variables  that  did 
not  have  statistically  significant  relationships  to  the  outcome  (or  dependent)  variable. 


Table  4-2.  Significant  Bivariate  Correlations. 


Variable 

Variable 

Correlation 

r  1  WL/Kis. 

ACINT 

PARED 

.089 

EDUCGOAL 

ACINT 

.008 

ACINT 

ABILITYH 

-.097 

ACINT 

FTWORK 

-.174 

SOCINT 

ABILITYH 

-.098 

SOCINT 

PARED 

.137 

PARED 

ABILITYH 

-.141 

PARED 

AGE 

-.229 

PARED 

NUMDEP 

-.221 

PARED 

OUTCOME 

.134 

PARED 

FTWORK 

-.095 

RISK 

FTWORK 

.488 

AGE 

FTWORK 

.146 

OUTCOME 

FTWORK 

-.172 

RISK 

ACINT 

-.186 

RISK 

SOCINT 

-.342 

RISK 

PARED 

-.299 

RISK 

AGE 

.616 

RISK 

NUMDEP 

.558 

FTWORK 

SOCINT 

-.198 

ACINT 

AGE 

-.129 

ACINT 

OUTCOME 

.196 

SOCINT 

AGE 

-.304 

SOCINT 

NUMDEP 

-.204 

SOCINT 

OUTCOME 

.209 

RISK 

OUTCOME 

-.259 

AGE 

OUTCOME 

-.156 

EDUCGOAL 

OUTCOME 

.123 

The  Resulting  Model 

The  results  of  the  logistic  regression  analysis  lead  to  an  identification  of  the 
variables  (of  those  variables  tested)  that  most  accurately  predicted  the  two-year  college 
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student  graduation  and  transfer  behavior  during  the  five  years  of  the  BPS:90/94  study. 
With  thirty  rejected  cases  due  to  missing  data,  the  number  of  cases  included  in  the 
analysis  was  674.  The  logistic  regression  model,  which  is  a  nonlinear  model,  resulted  in 
the  following  regression  equation  for  this  data: 

Probability  (OUTCOME)  =  1 

1+e" 

Where  z  =  -.6916  +  .4173(ACINT)  -  .2901(R1SK) 

Not  all  of  the  independent  variables  tested  were  used  in  the  regression  equation 
since,  as  is  the  case  with  multiple  linear  regression,  the  contribution  of  individual 
variables  in  logistic  regression  was  difficuh  to  determine.  The  inclusion  of  only  two 
independent  variables  in  the  model — the  ACINT  (academic  integration)  and  RISK 
(number  of  risk  characteristics  a  student  possessed)  variables — suggested  that  these  were 
the  only  two  variables  analyzed  that  could  explain  a  significant  portion  of  the  variation  in 
the  dependent  variable.  None  of  the  other  eight  independent  variables  analyzed  were 
included  in  the  resulting  model. 

> 

Interpreting  the  Logistic  Regression  Coefficients 

The  log  of  the  odds,  or  logit,  was: 

Logit==  log  (Prob  (event)/  Prob  (no  event))  =  p  +  px  +. . .  +p  x 

This  model  indicated  that  the  coefficient  of  the  RISK  variable  was  -.2901.  In 
other  words,  when  the  number  of  risk  variables  a  student  possessed  changed  fi-om  0  to  1, 
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and  the  values  of  the  other  independent  variables  remained  the  same,  the  log  odds  of  the 
outcome  variable  decreased  by  .2901.^ 

Exponentiated  Value  of  B  Coefficient 

The  exponentiated  values  of  the  B  coefficients,  Exp  (B),  are  the  effect  of  a  one- 
unit  change  in  an  independent  variable  on  the  log  odds  of  the  outcome;  this  is  also  called 
the  odds  ratio.  According  to  SPSS  Inc.,  exponentiated  values  of  the  B  coefficient  "can 
be  interpreted  as  estimates  for  the  effect  of  a  particular  variable,  controlling  for  the  other 
variables  in  the  equation."^  In  other  words,  the  odds  ratio  indicated  the  change  in  the 
odds  of  the  outcome  for  a  case  when  the  value  of  that  variable  increased  by  one. 

In  the  model  developed  by  this  research,  the  Exp  (B)  for  the  ACINT  variable 
indicated  that  when  a  student's  academic  integration  increased  (from  0  to  1  on  the 
ACINT  scale  used),  the  odds  of  him  graduating  or  transferring  within  the  five  years 
studied  were  increased  by  a  factor  of  1 .5 1 78.  For  the  RISK  variable,  the  odds  of 
graduation  or  transfer  decreased  by  .7482  as  the  RISK  variable  changed  from  0  to  1.'' 

Accuracy  of  Prediction 

"The  classification  of  cases  is  a  simple  matter  in  logistic  regression."^  The 
classification  table  shows  a  comparison  of  the  model's  predicted  outcomes  to  the  actual 
observed  outcomes.  The  classification  table's  ability  to  accurately  classify  cases  into  the 
two  categories  of  the  outcome  variable,  according  to  SPSS  Inc.,  is  a  "measure  of  how 
well  the  model  performs".' 

The  Maximum-likelihood  method,  which  is  analogous  to  the  Least-Squares 
Method  in  multiple  linear  regression  analysis,  resulted  in  the  inclusion  of  coefficients 
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and  variables  that  made  the  observed  results  of  the  sampled  students'  graduation  and 
transfer  behavior  most  "likely"  in  the  model.  In  general,  if  the  estimated  probability  was 
less  than  .5,  the  event  was  predicted  not  to  occur. 

In  order  to  classify  cases  into  the  two  possible  outcome  categories  (in  this  case, 
graduation/transfer  or  not  graduation/transfer)  a  case  is  predicted  to  be  in  the  lower  value 
of  the  outcome  variable  if  its  predicted  probability  is  less  than  .5  and  is  otherwise 
predicted  to  be  in  the  upper  category  of  the  outcome  variable.  In  this  case  (and  in 
standard  statistical  procedure),  the  classification  table  compared  the  observed  and 
predicted  group  memberships  when  cases  with  the  predicted  probability  of  .5  or  greater 
were  classified  as  graduating  or  transferring  within  five  years. '°  Table  4-3  is  the 
classification  table  for  the  logistic  regression  analysis. 

'  •  '■      Table  4-3.  Classification  Table. 

PREDICTED  OUTCOMES 


OBSERVED 
OUTCOMES 


0(=NOT 
GRAD  OR 
TRANS) 

1  (=  GRAD 
OR  TRANS) 


0(=  NOT  GRAD  OR 

1  (=  GRAD 

PERCENT 

TRANS) 

OR  TRANS) 

CORRECT 

258 

116 

68.98% 

142 

158 

52.67% 

OVERALL 


6L72% 


This  classification  table  indicated  that  the  model  correctly  predicted  the 
outcomes  of  258  students  who  did  not  graduate  or  transfer.  Also,  116  students  who  did 
not  graduate  or  transfer  were  incorrectly  predicted  to  graduate  or  transfer.  Similarly, 
142  students  who  did  graduate  or  transfer  were  incorrectly  predicted  not  to  graduate  or 
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transfer.  The  latter  group  represented  students  for  whom  the  community  college 
performed  well — these  students  who  possessed  qualities  known  to  hinder  postsecondary 
educational  success  were  able  to  graduate  or  transfer  within  five  years.  The  diagonal 
represented  the  cases  that  were  correctly  classified  while  the  off-diagonal  represented 
the  number  of  incorrectly  classified  cases.  *  ■ 

The  Percent  Correct  column  of  the  classification  table  showed  that  68.98%  of 
those  students  who  did  not  graduate  or  transfer  were  correctly  classified.  Of  those  who 
did  graduate,  52.67%  were  correctly  classified.  Overall,  61.72%  of  the  students  were 
correctly  classified.  '  •  '  ^  . 

Goodness  of  Fit  j 

A  model's  "goodness  of  fit"  to  the  data  is  one  way  to  analyze  the  strength  of  the 
model.  There  are  several  statistics  used  for  this  purpose,  which  follow: 

The  Wald  Statistic  - 

The  Wald  statistic,  which  is  the  square  of  the  ratio  of  the  coefficient  to  its 
standard  error,  was  15.87  and  34.87  for  the  ACINT  and  RISK  variables,  respectively. 
The  significance  levels  for  the  two  Wald  statistics  were  .0001  for  the  ACINT  variable 
and  .0000  for  the  RISK  variable.  Using  a  significance  level  of  0.05,  both  the  ACINT 
and  RISK  variables  were  significantly  different  from  zero.  In  other  words,  the  null 
hypothesis  that  the  ACINT  and  RISK  variables  were  equal  to  zero  was  rejected. 

Because  the  Wald  statistic  has  an  undesirable  property — mainly,  when  the 
absolute  value  of  the  regression  coefficient  becomes  large,  the  estimated  standard  error 
is  too  large— it  was  necessary  to  check  for  large  coefficients  that  would  result  in  failing 


to  reject  the  null  hypothesis 
model. 

R- Values 

An  R-value  is  one  way  to  test  the  strength  of  a  statistical  model.  The  R^,  which 
is  based  on  the  R-value,  indicates  the  percentage  of  the  variance  in  the  outcome  variable 
that  is  accounted  for  by  the  independent  variables  in  the  model.  Therefore,  as  the  R^ 
becomes  larger,  the  model  becomes  stronger.  The  R- values  for  the  ACINT  and  RISK 
variables  were  .1224  and  -.1 884  respectively,  reflecting  a  relatively  small  partial 
contribution  to  the  model.  . 

-2  Log  Likelihood  (-2LL) 

Another  way  to  assess  the  goodness  of  fit  is  to  examine  how  "likely"  the  sample 
results  actually  are,  given  the  parameter  estimates.  The  likelihood  is  the  "probability  of 
the  observed  results,  given  the  parameter  estimates"."  "It  is  customary  to  use  -2  times 
the  natural  log  of  the  likelihood  (-2LL)  as  a  measure  of  model  fit  since  it  has  ties  to  the 
chi-square  distribution."'^ 

The  -2LL  for  this  model  without  the  independent  variables  was  926.22.  < 
Although  small  values  of-2LL  indicate  a  good  model  with  high  likelihood  of  the 
observed  results,  "this  fit  statistic  is  not  usually  interpreted  directly".      A  comparison 
of  the  improvement  of  the  -2LL  with  the  independent  variables  added  to  the  model  is  a 
better  way  to  analyze  the  strength  of  the  model  than  is  the  -2LL  alone.  Although  a 
perfect  fit  would  result  in  a  -2LL  equal  to  zero,  once  the  RISK  and  ACINT  variables 
were  added  to  the  model  the  log  likelihood  improved  to  862.96.  The  difference  between 
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.  It  was  determined  that  this  was  not  the  case  with  this 
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the  -2LL  figures  (926.22-862.96)  results  in  the  Model  Chi-square  statistic  that  is 
covered  in  the  next  section  of  this  chapter. 

Model  Chi-Square 

The  Model  Chi-square  is  a  statistical  test  of  the  null  hypothesis  that  the 
coefficients  for  all  of  the  terms  in  the  model  are  zero.  The  difference  between  the  -2  LL 
in  the  model  without  the  two  independent  variables  and  the  -2  LL  in  the  model  with  the 
two  independent  variables  can  be  found  in  the  Model  Chi-square  column  of  the  SPSS 
output.  The  Model  Chi-square,  which  is  analogous  to  the  overall  F  test  for  regression, 
was  63.25  for  this  model. 

Nagelkerke's  Pseudo 

Nagelkerke's  pseudo  R  is  an  estimation  of  the  R  used  for  multiple  linear 
regression.  Although  there  are  other  estimations  of  the  linear  regression  R^, 
Nagelkerke's  pseudo  R^  is  preferred  because  it  can  achieve  a  maximum  value  of  one. 
According  to  this  Nagelkerke's  R^  only  about  1 2%  of  the  variation  in  the  outcome 

variable  was  explained  by  the  independent  variables  in  the  logistic  regression  model. 

• '  ■  .  " -  < '  >.  ^  ■ 

Predictive  Strength  Versus  Goodness  of  Fit 

According  to  SPSS  Inc.,  a  model's  goodness  of  fit  and  its  predictive  strength  do 
not  necessarily  correspond.  It  is  likely,  for  instance,  that  a  model  could  have  strong 
predictive  strength  while  lacking  goodness  of  fit.''*  Conversely,  "as  sample  size 
increases,  a  set  of  independent  variables  can  be  statistically  significant,  but  still  not  yield 
a  high  percentage  of  correct  predictions."'^  hi  this  research  the  predictive  strength  of  the 
model  was  of  greater  importance  than  was  the  significance  of  individual  variables  in  the 


model,  therefore,  more  emphasis  was  placed  on  the  classification  table  results  than  on 
the  goodness  of  fit  statistics  when  analyzing  the  model. 

A  Hypothetical  Example 

An  example  ofthe  utility  ofthis  model  follows.  The  following  model  (as 
reported  and  described  earlier  in  this  chapter)  was  developed: 

Probability  (OUTCOME)  =  1 


1+ 

Where  z  -  -.6916  +  .4173(ACINT)  -  2901(RISK) 

So,  the  application  of  hypothetical  individual  student  data  to  this  model,  for  a 
student  who  rates  a  1  on  the  academic  integration  scale  and  who  also  possesses  2  risk 
factors,  would  result  in  the  following  equation: 


Z  = -.6916 +  .4173(1) -.2901(2) 


Therefore,  the  probability  of  this  student  graduating  or  transferring  within  five 
years  was  estimated  as: 

Probability  (OUTCOME)  =  1 

=  .237 

1+/-.8545) 

Where  e  =  2.71 8  (the  base  of  the  natural  logarithm) 

In  general,  if  the  estimated  probability  of  the  outcome  (as  calculated  above)  is 
less  than  .5,  the  event  is  predicted  not  to  occur.  In  this  hypothetical  case,  the  model 
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predicts  that  this  student  would  not  graduate  or  transfer  because  the  calculated  estimated 
probability  of  the  outcome  equaled  .237  (which  is  less  than  the  .5  cut-off). 

Summary  of  Findings 

Overall,  the  output  for  this  logistic  regression  analysis  showed  that  two  variables, 
RISK  and  ACINT,  had  some  degree  of  predictive  power  for  the  graduation  and  transfer 
rates  of  this  group  of  two-year  and  community  college  students.  A  comparison  of  this  ' 
two-year  college  student  model  to  models  constructed  (by  other  researchers)  for  the 
four-year  college  sector  indicated  that  the  variables  that  were  predictive  of  student 
graduation  and  transfer  behavior  at  four-year  colleges  and  universities  were  different 
than  those  for  two-year  college  students.  The  model  was  able  to  predict  two-year  college 
students  who  would  not  graduate  or  transfer  with  68.98%  accuracy  and  those  who  would 
graduate  or  transfer  with  52.67%  accuracy.  Overall,  this  model  accurately  predicted 
61.72%  of  the  two-year  college  student  outcomes. 

Chapter  5  contains  the  discussion  of  the  results  of  the  study,  including  a 
summary  of  the  conclusions.  Implications  of  the  study  and  recommendations  for  further 
study  follow  the  summary. 

Notes 

'  Arlene  Fink,  How  to  Analyze  Survey  Data  (Thousand  Oaks:  Sage  Publications,  1995),  97. 

^  U.S.  Department  of  Education,  National  Center  for  Education  Statistics  (NCES),  1990-1994 
Beginning  Postsecondary  Students  Study  (BPS:  90/94). 

^  Alexander  W.  Astin,  Four  Critical  Years:  Effects  of  College  on  Beliefs,  Attitudes,  and 
Knowledge  (San  Francisco:  Jossey-Bass,  1977). 

"  SPSS,  Advanced  Techniques:  Regression  (Chicago:  SPSS,  January  6,  2000),  7-6. 

'  Because  of  its  negative  value,  this  would  result  in  a  decrease  rather  than  an  increase  in  value. 
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*  SPSS,  Advanced  Techniques:  Regression,  7-15. 

'  The  question:  "How  much  does  this  change  the  probability  of  the  outcome  for  this  individual?"  is 
less  straightforward,  however.  Because  the  probability  depends  on  where  that  individual  starts,  in  terms  of 
what  her  original  odds  of  graduating  or  transferring  were,  the  change  in  the  odds  ratio  could  be  very  small, 
very  large,  or  anywhere  in  between. 

*  SPSS,  Advanced  Techniques:  Regression,  7-5. 
'/Wrf,  7-14. 

According  to  SPSS,  a  cut-off  point  other  than  the  standard  default  of  .5  could  be  used,  however, 
it  was  not  recommended  unless  the  researcher  was  "very  experienced".  This  researcher  conducted 
exploratory  analyses  utilizing  cut-off  points  other  than  .5,  but  did  not  find  the  results  to  vary  significantly 
fi-om  the  results  gained  fi-om  using  the  .5  cut-off.  A  .5  default  cut-off  point  was  therefore  utilized. 
Ibid,  7-10. 

Ibid.,l-\\. 

SPSS,  Advanced  Techniques:  Regression,  7-11. 
^Ubid.  7-12. 
"  Ibid,  7-5. 
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CHAPTER  5 
CONCLUSIONS  AND  RECOMMENDATIONS 


Introduction 

The  purpose  of  this  study  was  to  examine  the  use  of  performance  assessment  and 
performance-based  funding  and  budgeting  (PBF&B)  in  United  States  community 
colleges.  This  study  was  designed  (1)  to  determine  whether  current  models  and 
performance  indicators  (Pis)  used  for  PBF&B  were  appropriate  for  assessing  the 
performance  of  two-year  postsecondary  institutions;  and,  if  not,  (2)  to  develop 
appropriate  Pis  through  the  introduction  of  a  predictive  model  of  community  college 
student  performance  which  could  be  used  to  assess  community  colleges. 

In  order  to  address  some  of  these  issues,  this  research  examined  the  following  two 
areas  of  inquiry: 

The  first  question  was  determining  how  performance  assessment  and  performance 
measures  were  used  to  allocate  state  funds  to  postsecondary  education  institutions.  Was 
performance  assessment  used  for  all  types  of  postsecondary  institutions— both  two-year 
and  four-year  colleges  and  universities?  Which  states  had  used,  were  using,  or  planned  to 
use  this  type  of  ftinding  allocation?  Of  the  states  currently  using  performance  funding  or 
budgeting,  what  portions  of  their  individual  budgets  did  they  allocate  in  this  manner? 
Did  states  use  multiple  measures  of  institutional  performance  to  allocate  funds,  and,  if  so, 
how  many  did  they  use?  Were  institutional  performance  measures  the  same  for  all 
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postsecondary  institutions?  Specifically,  was  performance  measured  differently  for  four- 
year  and  two-year  colleges  and  universities?  What  measures  of  postsecondary 
institutional  performance  did  states  most  frequently  use  to  allocate  funds? 

This  research  question  was  answered  through  a  review  of  the  literature  and  was 
addressed  in  Chapter  2. 

The  second  question  was  whether  the  institutional  performance  measures  used  by 
states  and  institutions  accurately  measured  the  performance  of  community  and  two-year 
colleges.  Could  the  outcome  measures  most  commonly  used  by  states  and  institutions 
accurately  depict  community  and  two-year  college  performance?  How  feasible  was  the 
development  of  performance  measures  to  account  for  differences  in  levels  and  types  of 
institutions?  Specifically,  was  there  a  single  model  that  could  serve  as  an  accurate 
performance  indicator  for  both  two-year  and  four-year  postsecondary  institutions? 

This  research  question  was  answered  through  a  review  of  the  literature  and 
through  the  use  of  logistic  regression  analysis  to  develop  a  model  of  community  college 
student  graduation  and  transfer  rates  and  subsequent  two-year  college  performance.  A 
review  of  the  literature  can  be  found  in  Chapter  2. 

The  purpose  of  the  logistic  regression  analysis  was  to  develop  a  model  that  could 
accurately  predict  community  college  student  graduation  and  transfer  behavior  at  two- 
year  institutions,  which  could  later  be  used  as  a  performance  assessment  tool  for  public 
two-year  colleges.  The  logistic  regression  analysis  and  resulting  model  offered  a  way  to 
account  for  the  student  characteristics  that  were  largely  out  of  institutional  control,  but 
which  contributed  to  the  likelihood  of  college  student  graduation  or  transfer  within  five 
years  (e.g.,  student  race,  educational  aspirations,  parental  level  of  education,  etc.). 
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An  accurate  model  that  utilized  incoming  student  characteristics  to  predict  college 
student  outcomes  could  later  be  compared  to  actual  student  graduation  and  transfer 
behavior  as  a  measure  of  institutional  performance.  In  other  words,  the  residual 
(predicted  versus  actual  student  graduation  and  transfer  rates)  could  offer  a  fair 
comparison  of  institutional  performance  across  colleges  and  universities.  This  analysis 
identified  some  of  the  variables  most  predictive  of  two-year  and  community  college 
student  graduation  and  transfer  behavior  by  eliminating  the  independent  variables 
selected  for  analysis  that  did  not  have  statistically  significant  relationships  to  the  outcome 
(or  dependent)  variable. 

Ten  independent  variables,  one  moderator  variable,  and  one  dependent  variable 
were  selected,  based  on  prior  research,  for  inclusion  in  the  logistic  regression  analyses. 
The  longitudinal  data  resulting  from  the  Beginning  Postsecondary  Students  1990-1994 
Study  (BPS:90/94)  conducted  by  the  National  Center  for  Education  Statistics  (NCES) 
was  utilized  for  the  regression  analyses.  The  results  of  the  analyses  are  included  in  this 
chapter  following  the  discussion  of  the  literature  review.  Further,  this  chapter  includes 
the  conclusions,  implications,  a  summary,  and  recommendations  for  further  study. 

Conclusions  and  Implications 

Performance  Assessment  for  the  Allocation  of  Funds 

Overall,  performance  funding  and  performance  budgeting  gained  popularity 
during  the  decade  of  the  1990s  as  a  credible  and  effective  means  of  funding  public  higher 
education.'  However,  the  growth  in  this  type  of  funding  generated  much  debate 
concerning  both  the  types  of  performance  indicators  (Pis)  utilized  to  reflect  higher 
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education  institutions'  quality  and  the  amount  of  state  budgets  to  be  allocated  in  this 
manner.   Most  states  that  utilized  performance  funding  or  performance  budgeting  did  not 
include  a  separate  set  of  indicators  for  their  community  colleges.^  This  lack  of 
differentiation  between  the  indicators  for  four-year  and  two-year  colleges  created  the 
potential  for  inequitable  distribution  of  funds  to  higher  education/ 

According  to  the  literature,  a  model  that  could  accurately  predict  the  success  of 
entering  college  students  (specifically,  graduation  or  transfer  rates  from  a  higher 
education  institution)  could  be  compared  to  actual  success  rates  at  individual  institutions 
as  a  measure  of  institutional  performance.^  Although  this  type  of  model  was  developed 
successfully  for  other  purposes  in  the  literature,  it  could  be  extended  to  the  performance- 
funding  arena. 

Performance-based  funding  and  budgeting  (PBF&B)  research  focused  mainly  on 
the  public  four-year  postsecondary  education  sector.  Public  two-year  college  literature, 
although  scarce,  indicated  that  few  states  utilized  performance  indicators  developed 
specifically  for  the  two-year  college  sector.  The  most  commonly  used  performance 
indicators  in  the  states  were  absolute  measures  of  graduation  and  transfer  rates. 

According  to  the  literature,  public  two-year  colleges  had  lower  graduation  and 
transfer  rates  than  did  the  public  four-year  college  and  university  sector.  The  literature 
pointed  to  several  contributing  factors  that  explained  some  of  the  lower  graduation  and 
transfer  rates  at  two-year  institutions.  Overall,  two-year  colleges  were  attended  by 
students  with  weaker  academic  preparation,  who  were  more  likely  to  work  full  time,  who 
had  a  broader  range  of  academic  goals  and  purposes  for  attending  college,  who  were 
more  likely  to  have  financial  dependents  and  to  be  single  parents,  who  attended  college 
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on  a  part-time  basis,  and  who  had  delayed  their  college  enrollment  by  more  than  a  year 
after  high  school  graduation. 

The  differences  in  the  types  of  students  enrolled  at  two-year  and  four-year 
institutions  suggested  that  absolute  indicators  of  institutional  performance  used  to  assess 
both  types  of  institutions  (such  as  graduation  and  transfer  rates)  did  less  to  indicate  the 
quality  of  institutions  than  they  did  to  describe  the  type  of  students  enrolled  at  these 
institutions.  The  literature  suggested  the  development  of  a  performance  measure  that 
could  account  for  some  of  the  differences  in  student  characteristics.  Such  a  measure 
could  offer  a  reference  norm  that  would  make  the  comparison  of  all  institutions  in  a  state 
(both  four-year  and  two-year)  more  accurate,  thus  improving  the  equity  of  PBF&B  within 
state  systems. 

A  norm-referenced  performance  indicator  could  be  developed  through  a 
multivariate  regression  analysis  using  incoming  student  characteristics  as  the  independent 
variables  and  graduation  or  transfer  as  the  dependent  variable.  A  model,  which  resulted 
from  the  logistic  regression  analysis,  could  accurately  predict  two-year  college  student 
graduation  and  transfer  behavior  based  solely  on  incoming  student  characteristics.  A 
predicted  graduation  or  transfer  rate  for  an  institution  could  then  be  compared  to  actual 
rates  as  a  measure  of  performance. 

The  lack  of  a  nationally  representative  longitudinal  database  of  public  two-year 
college  students  was  the  greatest  barrier  to  developing  a  norm-referenced  performance 
indicator  for  community  colleges.  Because  the  BPS:90/94  was  the  first  study  that 
included  a  national  sample  of  all  beginning  postsecondary  students,  it  offered  the  means 
for  developing  a  norm-referenced  performance  indicator. 
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Analysis  of  Logistic  Regression  Results 

The  logistic  regression  analysis  addressed  the  second  area  of  research  inquiry  in 
this  study.  For  the  purpose  of  this  research,  it  was  believed  that  four-year  college 
variables  discovered  by  previous  researchers  might  be  appropriate  for  the  prediction  of 
two-year  student  graduation  and  transfer  rates.  Because  four-year  college  student  studies 
were  the  only  prior  research  conducted  of  this  kind,  they  would  provide  a  starting  point 
for  two-year  college  student  predictions. 

Based  on  prior  research  conducted  about  the  four-year  sector,  it  was  assumed  that 
the  following  ten  variables  would  affect  graduation  and  transfer  rates  of  two-year 
students  in  the  following  way: 

1 .  The  RISK  variable  was  assumed  to  have  a  negative  relationship  with  the 
outcome  variable,  OUTCOME,  because  it  was  believed  that  the  more 
known  risk  factors  a  student  possessed  the  less  likely  that  student  was  to 
graduate  or  transfer  from  college  within  five  years. 

2.  The  academic  integration  of  a  student,  ACINT,  was  believed  to  have  a 
positive  relationship  with  the  OUTCOME  variable  because  this  variable 
should  measure  the  degree  to  which  a  student  was  integrated  with  the 
academic  aspects  of  his  college. 

3.  The  SOCINT  variable,  the  degree  to  which  a  student  was  socially 
integrated  with  her  college,  was  assumed  to  have  a  positive  relationship 
with  the  OUTCOME  variable. 

4.  A  student  who  worked  full  time,  measured  by  the  FTWORK  variable,  was 
assumed  to  have  a  negative  relationship  with  the  OUTCOME  variable 
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because  a  student  with  a  full-time  job  was  believed  to  have  less  time  to 
concentrate  on  college  studies. 

5.  A  student's  educational  goals,  as  measured  by  the  EDUCGOAL  variable, 
was  believed  to  have  a  positive  relationship  with  the  OUTCOME  variable 
because  the  higher  numbers  on  the  scale  represented  the  higher 
educational  aspirations  of  a  student — a  student  who  did  not  intend  to 
graduate  or  transfer  would  be  less  likely  to  do  so. 

6.  The  AGE  of  a  student  was  assumed  to  have  a  negative  relationship  with 
the  OUTCOME  variable  because  prior  research  found  that  the  longer  a 
student  delayed  college  enrollment  after  high  school  graduation,  the  less 
likely  that  student  was  to  continue  on  to  college  graduation. 

7.  The  level  of  parental  education,  measured  by  the  PARED  variable,  was 
assumed  to  have  a  negative  relationship  with  the  OUTCOME  variable 
because  prior  research  indicated  that  students  coming  from  households 
where  neither  parent  had  attended  college  were  less  likely  to  graduate 
from  college. 

8.  A  student's  religion,  as  indicated  by  the  RELIGION  variable,  was 
assumed  to  have  a  negative  relationship  with  the  OUTCOME  variable 
because  prior  research  indicated  that  Catholic  and  Jewish  students  were 
more  likely  to  graduate  from  college  than  were  non-Catholic  and  non- 
Jewish  students. 

9.  The  number  of  financial  dependents  a  student  possessed,  NUMDEP,  was 
assumed  to  have  a  negative  relationship  with  the  OUTCOME  variable 
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because  it  was  believed  that  the  more  people  for  which  a  student  was 
financially  responsible,  the  less  time  the  student  would  have  to  devote  to 
college  studies. 

10.      The  student's  perception  of  his  academic  abilities,  ABILITYH,  was 
assumed  to  have  a  positive  relationship  with  the  OUTCOME  variable. 
This  assumption  was  based  on  prior  research  that  found  that  a  student's 
high  school  grade  point  average  (GPA)  was  highly  predictive  of  a 
student's  likelihood  of  graduation  from  college.  Because  the  ABILITYH 
variable  was  seen  as  a  proxy  for  a  student's  GPA  it  was  assumed  that  it, 
too,  would  have  a  strong  positive  relationship  with  a  student's  likelihood 
to  graduate  from  college  or  to  transfer  to  a  four-year  college  within  five 
years. 

The  above  assumptions  regarding  the  relationships  between  eight  of  the  ten 
independent  variables  and  the  outcome  variable  were  confirmed  by  this  research.  Two  of 
the  ten  independent  variables  did  not  show  significant  correlations  with  the  dependent 
variabl^RELIGION  and  NUMDEP  (number  of  dependents). 

The  correlation  analysis  conducted  prior  to  the  logistic  regression  indicated  that 
the  following  independent  variables  had  statistically  significant  relationships  with  the 
dependent  variable  (at  the  .05  alpha-level):  ACINT,  SOCINT,  RISK,  AGE, 
EDUCGOAL,  PARED,  ABILITYH,  and  FTWORK. 

The  lack  of  statistically  significant  relationships  between  two  of  the  independent 
variables  and  the  outcome  variable  may  be  due  to  the  fact  that  they  were  not  strongly 
related  to  the  graduation  or  transfer  behavior  of  community  college  students.  Other 
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possible  explanations  for  the  lack  of  statistically  significant  relationships  include  the 
measurement  error  associated  with  collecting  data  from  students  (as  opposed  to  more 
reliable  student  academic  transcript  data)  or  coding  errors  associated  with  data  entry  and 
similar  activities. 

The  possibility  that  a  student's  religion  and  number  of  dependents  were  more 
strongly  related  to  the  graduation  and  transfer  behavior  of  four-year  students  than  they 
were  to  two-year  college  students  may  actually  be  due  to  the  greater  diversity  of  students 
(in  terms  of  these  two  factors)  at  two-year  colleges.  The  degree  of  academic  and  social 
integration  of  a  student  may  increase  as  a  student  perceives  herself  as  being  similar  to 
other  students  in  her  classes.  Increases  in  integration  may  result  in  an  increased 
likelihood  of  a  minority  student  graduating  of  transferring  from  a  two-year  institution. 
So,  although  a  four-year  college  student's  religion  and  number  of  dependents  may  be 
predictive  of  her  likelihood  of  college  graduation  and  transfer,  they  appeared  to  have  less 
significance  in  this  two-year  college  study. 

The  logistic  regression  analyses  resulted  in  a  model  that  included  only  two  of  the 
ten  independent  variables  tested.  Such  a  situation  suggested  that  (of  the  ten  independent 
variables  included  in  this  research)  only  the  RISK  and  ACINT  variables  were  predictive 
of  the  graduation  and  transfer  behavior  of  two-year  college  students  (as  opposed  to  those 
variables  merely  being  related  to  graduation  and  transfer  behavior).  The  predictive 
strength  of  the  model  also  suggested  that  a  stronger  model  could  be  developed  with 
independent  variables  that  had  more  influence  on  the  transfer  and  graduation  behavior  of 
community  college  students. 
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It  seems  likely  that  there  are  other  variables  that  were  not  included  in  this 
regression,  which  might  improve  the  predictive  strength  of  this  model.  For  instance, 
although  high  school  grade  point  average  (GPA)  data  was  not  collected  in  the  BPS:90/94 
study,  this  data  is  thought  to  be  highly  predictive  of  student  success  in  college.  In  fact, 
GPA  has  been  utilized  as  common  admissions  criteria  by  postsecondary  institutions  for 
decades.  Unfortunately,  because  of  the  high  percentage  of  two-year  college  students  who 
postpone  college  attendance  for  more  than  a  year  after  high  school  graduation,  GPA  may 
not  be  as  accurate  at  predicting  two-year  college  student  success  as  it  is  in  the  four-year 
college  sector. 

Standardized  test  scores  from  tests  such  as  the  SAT  and  the  ACT  are  also  useful 
predictors  of  college  student  success.  In  conjunction  with  high  school  GPA,  SAT  and 
ACT  scores  are  commonly  used  to  predict  college  student  success  for  college  and 
university  admissions  purposes.  Unfortunately,  because  of  the  open-door  policy  at 
community  colleges  (i.e.,  the  lack  of  admissions  requirements)  most  community  college 
students  do  not  take  the  SAT  nor  the  ACT. 

Overall,  it  appeared  that  although  eight  of  the  independent  variables  were 
correlated  with  the  outcome  variable  (at  the  .05  alpha-level),  the  regression  analysis 
indicated  that  there  might  be  other  variables  that  could  more  accurately  predict  two-year 
college  student  graduation  and  transfer  rates.  Because  the  independent  variables  were 
selected  based  on  four-year  student  research,  it  appeared  that  the  variables  selected  might 
not  be  adequate. 

The  results  of  the  bivariate  correlation  and  logistic  regression  analyses  suggested 
that  predictors  of  two-year  and  four-year  college  student  graduation  and  transfer  behavior 
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differed.  This  finding  supported  the  belief  that  more  longitudinal  data  should  be 
collected  about  two-year  college  students.  Without  such  data,  it  is  unlikely  that  a  flexible 
institutional  performance  indicator  can  be  developed  for  use  in  both  the  four-year  and 
two-year  college  sectors.  Consequently,  without  a  flexible  performance  indicator  that 
can  accurately  assess  the  performance  of  two-year  colleges,  these  institutions  will 
continue  to  be  subjected  to  the  standards  of  four-year  institutions — standards  that  are 
inappropriate  for  use  in  the  two-year  college  sector. 

As  states  continue  to  adopt  performance-based  funding  and  budgeting  schemas 
for  the  allocation  of  funds  to  postsecondary  education,  they  should  continue  to  search  for 
ways  to  fairly  and  adequately  fund  all  sectors  of  higher  education.  An  approach  that 
takes  into  account  the  differences  in  the  populations  served  by  the  various  postsecondary 
sectors  would  offer  a  way  to  reward  institutions  for  their  academic  and  managerial 
practices  instead  of  rewarding  them  for  the  type  of  student  they  enroll. 

Summary 

In  summary,  the  literature  revealed  that  performance  assessment  and  performance 
measures  were  used  to  allocate  state  funds  to  postsecondary  education  institutions  for  all 
types  of  postsecondary  institutions — both  two-year  and  four-year  colleges  and 
universities.  The  majority  of  the  states  had  used,  were  using,  or  planned  to  use  this  type 
of  funding  allocation,  but  most  states  preferred  to  allocate  only  small  portions  of  their 
funds  in  this  manner.  Although  most  states  used  multiple  measures  of  institutional 
performance  to  allocate  funds,  graduation  and  transfer  rates  were  used  in  nearly  all  of  the 
states.  In  most  cases  institutional  performance  measures  were  the  same  for  all 
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postsecondary  institutions  within  a  state,  which  meant  that  performance  was  not 
measured  differently  for  four-year  and  two-year  colleges  and  universities. 

Results  from  the  literature  review  provided  justification  for  the  development  of  a 
flexible  performance  indicator  that  could  account  for  student  characteristics  when 
assessing  institutional  performance.  The  review  of  the  literature  also  provided  data  and 
suggestions  for  the  variables  included  in  the  logistic  regression  analyses.  Several  student 
characteristics  found  to  be  influential  in  the  four-year  sector  appeared  to  be  appropriate 
predictors  for  the  two-year  college  sector — ^although  they  were  not  completely  adequate. 

Recommendations  for  Future  Study 

Improvement  of  the  model  may  include  the  addition  of  other  variables  from  the 
Beginning  Postsecondary  Students  1990-1994  study  in  the  logistic  regression  analysis. 
New  data  from  the  National  Center  for  Education  Statistics  (NCES)  also  includes  a 
follow-up  to  the  original  BPS:90/94  study  and  may  provide  data  for  the  improvement  of 
the  model. 

More  in-depth  analyses  of  the  data  used  in  this  research  may  also  prove  useful. 
Specifically,  testing  the  components  of  the  constructed  RISK  variable  while  excluding 
the  actual  RISK  variable  may  result  in  a  model  that  better  fits  the  data.  Also,  since  the 
variables  used  in  these  analyses  were  selected  on  the  basis  of  prior  research  conducted 
about  four-year  colleges  and  students,  it  is  necessary  to  explore  variables  not  yet  used  in 
this  type  of  research. 

Another  tactic  for  improving  the  prediction  of  graduation  and  transfer  rates  may 
be  to  use  institutional  data  obtained  from  ACT,  Inc.^  The  data  collected  by  ACT,  Inc. 
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may  result  in  a  better  fit  between  the  data  and  the  model.  The  anticipated  result  would  be 
a  more  accurate  prediction  of  the  outcomes  of  concern. 

Once  a  model  is  developed  that  can  adequately  predict  the  graduation  and  transfer 
behavior  of  two-year  college  students,  the  next  step  will  be  to  apply  institutional  data  to 
the  new  model.  This  application  will  result  in  the  prediction  of  student  graduation  and 
transfer  behavior  at  various  institutions,  which  can  then  be  compared  to  actual  graduation 
and  transfer  rates  at  those  same  institutions.  Because  the  necessary  institutional  data  for 
community  colleges  has  been  found  in  Florida,  this  state  is  a  good  candidate  for  the 
analyses. 


Notes 

'  Joseph  C.  Burke,  "Performance  Budgeting  and  Funding:  Making  Results  Count  Where  It 
Maners"  Assessment  Update  10,  no  .6  (November/December  1998):  1. 

^  See  National  Center  for  Research  in  Vocational  Education,  The  Returns  to  Education  and 
Training  in  the  Sub-Baccalaureate  Labor  Market:  Evidence  from  the  Survey  of  Income  and  Program 
Participation  1984-1990  (Berkley:  University  of  California);  and  Community  College  Roundtable, 
Community  Colleges:  Core  Indicators  of  Effectiveness:  A  Report  of  the  Community  College  Roundtable 
(Washington,  DC:  American  Association  of  Community  Colleges,  National  Center  for  Higher  Education, 
1994.) 

'  Sandra  Ruppert,  Focus  on  the  Customer:  A  New  Approach  to  State-Level  Accountability 
Reporting  and  Processes  for  Higher  Education  (SHEEO:  July  1998),  3. 

See  Community  College  Roundtable,  Core  Indicators  of  Effectiveness;  Andreea  Serban,  "The 
Performance  Funding  Wave:  Views  of  State  Policymakers  and  Campus  Leaders,"  Assessment  Update  10, 
no.  2  (March/April  1998);  and  Harriott  Calhoun,  "Implementing  Institutional  Effectiveness  at  Two- Year 
Colleges,"  in  A  Practitioner 's  Handbook  for  Institutional  Effectiveness  and  Student  Outcomes  Assessment 
Implementation,  3"*  ed.  (Edison,  N.J.:  Agathon  Press,  1995),  223. 

'  See  Jill  Johnes  and  Jim  Taylor,  "An  Evaluation  of  Performance  Indicators  Based  upon  the  First 
Destination  of  University  Graduates,"  Studies  in  Higher  Education  14,  no.  2  (1989):  201-217;  Alexander 
W.  Astin,  Predicting  Academic  Performance  in  College:  Selectivity  Data  for  2300  American  Colleges 
(New  York:  Free  Press,  1971);  and  Thomas  G.  Mortenson,  Postsecondary  Education  Opportunity 
(Mortenson  Research  Seminar,  April  1997). 

*  ACT,  Inc.  is  an  independent,  nonprofit  organization  that  provides  educational  services  to 
students  and  their  parents,  to  high  schools  and  colleges,  to  professional  associations  and  government 
agencies,  and  to  business  and  industry.  ACT's  ASSET  paper  and  pencil  test  was  administered  to  over 
775,000  students  at  more  than  500  two-year  colleges  in  the  academic  year  1998-99. 
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